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Introduction 


The  goal  of  the  phase  II  of  the  project  is  to  prospectively  test  the  methods  and  techniques  developed 
in  phase  I  of  the  project.  Evidence-based  Clinical  Decision  Support  system  seeks  to  elicit  patients' 
preferences  and  aid  with  prognostication  of  the  life  expectancy  in  terminally  ill  patients  to  improve 
referral  of  patients  to  hospice  at  the  point  of  care.  In  addition,  the  EBM-CDSS  will  be  expanded  with 
an  evidence  based  pain  management  module  (EB-PMM)  to  assist  physicians  managing  patients  with 
pain. 

Body: 

Key  research-related  accomplishments  (since  the  submission  of  previous  annual  progress  report): 

•  We  completed  training  and  submitted  the  required  documents  for  our  nurses  and  other  research 
personnel  to  complete  TGHs'  credentialing  procedures.  This  step  was  essential  to  initiate  the 
prospective  phase  of  our  study. 

•  We  revised  our  case  report  forms  and  the  EBM-CDSS  software  including  its  graphic  user  interface. 

•  We  also  revised  the  user  guide  for  the  EBM-CDSS  software  and  the  training  manual  to  assist  research 
team  for  data  collection  in  the  prospective  phase  of  the  study.  We  requested  our  research  nurses  and 
other  research  personnel  to  use  the  EBM-CDSS  and  review  the  user  guide.  Based  on  their  feedback  we 
revised  the  software  and  the  user  guide  and  again  requested  them  to  review  it.  After  at  least  4 
iterations  we  have  finalized  EBM-CDSS  software  and  user  guide  (see  appendix  11).  (NB  there  is 
continuing  quality  monitoring;  depending  on  the  feedback  and  the  "situation  on  the  grounds",  we 
continue  to  revise  and  adjust  our  software  while  retaining  the  fidelity  toward  the  study  goals). 

•  We  invested  significant  amount  of  time  in  training  the  research  nurses  in  using  the  EBM-CDSS 
software  and  fine  tuning  their  interviewing  skills.  We  conducted  a  number  of  mock  interview  sessions 
in  which  our  research  nurses  conducted  interviews  using  the  EBM-CDSS  software,  accompanying  data 
collection  forms  (appendix  8),  scripts  (see  appendix  9)  and  informed  consents  (appendix  10). 

•  Our  research  nurses  have  also  completed  their  TGH  electronic  record  system  (EPIC)  training. 

•  We  have  also  completed  a  number  of  administrative  requirements  such  as  obtaining  a  University  of 
South  Florida  (USF)  Email  IDs,  TGFI  identification  cards,  parking  permits  and  USF  business  cards  for  our 
research  nurses.  (Although  these  steps  may  appear  small,  in  reality  they  are  all  time-consuming 
aspects  of  research  activities,  particularly  because  almost  all  modifications  to  the  protocol  require  the 
USF  IRB  approval). 

•  We  have  also  finalized  the  study  advertisement  flyer  including  obtaining  related  administrative 
permissions  to  advertise  our  study  at  TGFI  and  among  USF  physicians.  That  is,  we  have  obtained 
permissions  from  USF  media  relations  department  to  use  the  "USF  Flealth"  logo  on  our  flyer  and 
clearance  from  TGFI  to  post  our  flyer  across  TGFI  campus.  See  the  flyer  attached  in  appendix  12. 

•  We  conducted  a  number  of  meetings  with  TGFI  palliative  care  team  and  presented  our  research  study 
to  the  TGFI  palliative  care  team.  These  meeting  were  fruitful;  especially  in  establishing  trust  and 
working  relationship  with  TGFI  palliative  care  team,  which  is  a  key  to  facilitate  the  patients  referral  to 
our  study.  We  are  pleased  that  Dr.  Floward  Tuch,  director  of  the  USF  TGFI  team  joined  our  study  as  a 
key  clinical  collaborator. 
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We  have  met  with  the  number  of  the  key  physicians  and  their  teams  to  explain  the  purpose  of  the 
study.  This  will  be  continuing  effort  on  our  part  as  the  study  cannot  succeed  without  adequate 
referrals  from  the  physicians  at  TGH. 

Based  on  the  feedback  received  from  health  providers,  including  some  patients  and  families,  we  have 
further  revised  and  fine-tuned  our  CDSS: 

o  Revised  the  user  guide  for  the  EBM-CDSS  software  and  the  training  manual  to  assist  research 
team  for  data  collection  in  the  prospective  phase  of  the  study, 
o  Revised  the  script  to  guide  the  patient/proxy/provider  interview, 
o  Completed  further  training  of  research  nurses  regarding  EBM-CDSS  software  and  data 
collection  procedures. 

o  Further  developed  and  completed  internal  testing  of  the  pain  module  of  the  EB-PMM  module. 
We  have  refined  our  EB-PMM  to  complement  the  EBM-CDSS.  Our  objective  is  to  develop  a  reliable 
dosage  conversion  system  as  well  as  a  knowledge  based  for  each  available  pain  medication.  We  have 
also  incorporated  evidence  profiles  for  each  drug  to  support  the  decision  making  using  our  pain 
management  module.  We  have  also  created  a  survey  to  test  usefulness  of  EB-PMM  its  users. 

Recently,  the  National  Comprehensive  Cancer  Network  revised  their  guidelines  for  management  of 
adult  cancer  pain  and  we  have  revised  EB-PMM  accordingly.  We  have  contacted  the  TGH  palliative 
care  team  to  review  EB-PMM  and  provide  feedback  to  us.  Please  see  the  user  manual  for  EB-PMM 
and  the  EB-PMM  evaluation  survey  in  appendix  13  and  14  respectively. 

All  these  activities  have  been  necessary  to  help  us  open  the  patient  enrollment  in  the  prospective 
phase  of  the  study. 

Unfortunately,  as  we  were  to  start  enrollment,  our  research  nurses  left  our  study  team.  So,  we  are  in 
the  process  of  hiring  new  research  nurses.  We  have  already  identified  highly  qualified  candidates  to 
replace  the  existing  research  team.  Although  we  will  have  to  train  new  research  team  again,  the 
experience  gained  in  the  process  outline  above,  will  have  proven  invaluable  to  speed  up  training. 

We  believe  that  we  have  been  highly  productive  and  we  are  on  the  right  track  to  achieve  of  the 
objectives  of  the  study.  This  is  best  reflected  in  our  publication  records  (see  next  session).  As  it  can  be 
seen,  these  analyses  were  key  to  informing  the  prospective,  phase  II  of  our  project. 
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Reportable  outcomes 

We  have  published/submitted  7  manuscripts  for  publication  in  peer-reviewed  journals  (see  appendix 
1-7):  specifically: 

•  Manuscript  1  (appendix  1): 

Title:  Towards  a  Classification  Model  to  Identify  Hospice  Candidates  in  Terminally  III  Patient 
In  brief:  A  Rough  Set  Theory  based  classification  model  to  identify  hospice  candidates  within  a 
group  of  terminally  ill  patients.  Unlike  traditional  data  mining  methodologies,  this  approach 
using  an  artificial  methodology  of  rough  set  theory  seeks  to  identify  subgroups  of  patients 
possessing  common  characteristics  that  distinguish  them  from  other  subgroups  in  the 
dataset. 

Citation:  Eleazar  Gil-Herrera,  Ali  Yalcin,  Athanasios  Tsalatsanis,  Laura  E.  Barnes  and  Benjamin 
Djulbegovic,  "Towards  a  Classification  Model  to  Identify  Hospice  Candidates  in  Terminally  III 
Patients",  to  appear  in  the  Proceedings  of  the  Annual  International  Conference  of  the  IEEE 
Engineering  in  Medicine  and  Biology  Society,  2012 

•  Manuscript  2  (appendix  2): 

Title:  A  Flexible  Alternative  to  the  Cox  Proportional  Hazards  Model  for  Assessing  the 
Prognostic  Accuracy  of  Hospice  Patient  Survival 

In  brief:  We  applied  the  Palliative  Performance  Scale  to  demonstrate  that  a  novel  survival 
model  predicted  hospice  patient  survival  more  closely  than  a  commonly  used  Cox 
proportional  hazards  model.  Unlike  the  Cox  proportional  hazards  model,  it  uses  splines  and 
the  probit  link  function  to  model  patient  baseline  survival.  This  model  will  be  integrated  in 
the  prospective  phase  of  the  project. 

Citation:  Miladinovic  B,  Kumar  A,  Mhaskar  R,  Kim  S,  Schonwetter  R,  Djulbegovic  B,  A  flexible 
alternative  to  the  Cox  proportional  hazards  model  for  assessing  the  prognostic  accuracy  of 
patient  survival.  PLoS  ONE  2012(in  press) 

•  Manuscript  3  (appendix  3): 

Title:  Natural  History  of  Patients  With  Lung  Cancer  Without  Treatment:  A  systematic  Review. 
This  manuscript  is  currently  under  review  by  Supportive  Care  in  Cancer  journal. 

In  brief:  The  purpose  of  this  study  was  to  conduct  a  systematic  review  and  meta-analysis  of 
the  natural  history  of  patients  with  confirmed  diagnosis  of  NSCLC  without  active  treatment. 
Studies  were  identified  by  search  of  electronic  databases  (MEDLINE  and  CENTRAL)  and 
abstract  proceedings.  Data  on  mortality  was  extracted  from  all  included  studies  and  pooled 
proportion  of  mortality  was  calculated.  The  meta-analysis  included  seven  cohort  studies 
(4,418  patients)  and  15  randomized  controlled  trials  (1,031  patients).  The  pooled  proportion 
of  mortality  without  treatment  in  cohort  studies  was  0.97  (95%  Cl:  0.96  to  0.99)  and  0.96  in 
randomized  controlled  trials  (95%  Cl:  0.94  to  0.98)  over  median  study  periods  of  8  and  3 
years,  respectively.  There  was  no  statistically  significant  difference  between  these  subgroups. 
For  NSCLC  patients  undergoing  no  treatment,  mortality  is  very  high.  Although  limited  by  study 
design,  this  finding  provides  the  basis  for  future  trials  to  determine  optimal  expected 
improvement  in  mortality  with  innovative  treatments.  The  results  will  also  inform  the  aspects 
of  phase  II  part  of  our  studies  related  to  cancer  patients. 

Citation:  Wao,  H.,  Mhaskar,  R.,  Kumar,  A.,  &  Djulbegovic,  B.  Natural  history  of  patients  with 
lung  cancer  without  treatment:  A  systematic  review.  Journal  of  Supportive  Care  in  Cancer, 
2012  (under  review). 

•  Manuscript  4  (appendix  4): 
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Title:  External  validation  of  a  web-based  prognostic  tool  for  predicting  survival  in  patients  in 
hospice  care 

In  brief:  This  study  was  undertaken  to  assess  whether  another  prognostic  model  (a  popular 
model  Prognostat)  should  be  integrated  in  the  phase  II  of  our  project.  Prognostat  is  an 
interactive  Web-based  prognostic  tool  for  estimating  hospice  patient  survival  based  on  a 
patient's  palliative  performance  scale  score,  age,  gender,  and  cancer  status.  The  tool  was 
developed  using  data  from  5,893  palliative  care  patients  collected  at  the  Victoria  Hospice 
since  1994.  This  study  externally  validates  Prognostat  on  a  retrospective  cohort  of  590 
hospice  patients  in  Lifepath  Hospice  Center,  Florida,  USA.  The  criteria  used  to  evaluate  the 
prognostic  performance  are  the  Brier  score,  area  under  the  receiver  operating  curve, 
discrimination  slope  and  Hosmer-Lemeshow  goodness-of-fit  test.  Though  the  Kaplan-Meier 
curves  show  each  PPS  level  to  be  distinct  and  significantly  different,  the  findings  reveal  poor 
performance  of  the  prognostic  tool  in  our  cohort  of  hospice  patients.  Before  redeveloping  a 
new  prognostic  model  researchers  are  encouraged  to  combine  survival  estimates  obtained 
using  Prognostat  with  the  information  from  their  cohort  of  patients.  To  that  end,  Prognostat 
needs  to  explicitly  report  patient  risk  scores  to  be  useful  to  clinicians.  Hence,  we  will  not  use 
it  in  a  prospective  phase  of  the  project. 

Citation:  Miladinovic  B,  Mhaskar  R,  Kumar  A,  Kim  S,  Schonwetter  R,  Djulbegovic  B.  External 
validation  of  a  web-based  prognostic  tool  for  predicting  survival  in  patients  in  hospice  care. 
Journal  of  Palliative  Care,  2012  (submitted). 

Manuscript  5  (appendix  5): 

Title:  External  Validation  of  Prognostic  Models  in  Terminally  III  Patients. 

In  brief:  In  the  phase  I  of  our  project  we  retrospectively  extracted  data  from  590  deceased 
patients  enrolled  in  Tampa  Bay  Lifepath  Hospice  and  Palliative  Care  starting  January  2009  and 
going  backwards  to  validate  following  5  prognostic  models  commonly  recommended  in  the 
literature  for  prognostication  in  terminally  ill  patients:  1)  declining  exponential  approximation 
of  life  expectancy  (DEALE)  2)  study  to  understand  prognoses  and  preferences  for  outcomes 
and  risks  of  treatments  (SUPPORT),  3)  adjusted  palliative  performance  scale  (PPS),  4)  adjusted 
Karnofsky  performance  scale  index  (Karnofsky)  and  5)  adjusted  eastern  cooperative  oncology 
group  performance  status  (ECOG). 

The  models  were  tested  against  observed  survival  duration.  We  utilized  several  metrics  to 
assess  the  performance  of  these  models.  Specifically,  we  used  the  Brier  score  and  scaled  Brier 
score  (which  is  very  similar  to  the  Pearson  correlation  coefficient  R2),  the  area  under  the 
receiver  operating  characteristic  curve  (AUROC),  and  the  Hosmer-Lemshow  goodness-of-fit  p- 
value  (HL).  Based  on  our  analysis  we  will  use  PPS  and  SUPPORT  models  for  prognostication  of 
life  expectancy  in  the  prospective  phase  of  the  study 

Citation:  Mhaskar  R,  Miladinovic  B,  Tsalatsanis  A,  Mbah  A,  Kumar  A,  Sehwan  K,  Schonwetter  R, 
Djulbegovic  B.  External  Validation  of  Prognostic  Models  in  Terminally  III  Patients.  Blood;  1791:  4186, 
2011 

Manuscript  6  (appendix  6): 

Title:  Rough  Set  Theory  based  Prognostic  Model  for  Hospice  Referral 

In  brief:  Objective:  The  goal  of  this  paper  is  to  provide  an  accessible  prognostic  classification 
model  for  hospice  referrals.  Hospice  care  provides  high-quality  and  cost-effective  end-of-life 
care  for  terminally  ill  patients.  In  addition,  the  paper  explores  the  application  of  Rough  Set 
Theory  (RST)  for  the  development  of  clinically  credible  prognostic  models. 

Methods:  We  utilize  retrospective  data  from  9,103  terminally  ill  patients  to  demonstrate  the 
design  and  implementation  of  a  classifier  based  on  RST  for  potential  hospice  candidates.  RST 
provides  methods  for  knowledge  reduction,  founded  on  the  relational  indiscernibility  of 
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objects  in  a  decision  system,  to  describe  required  conditions  for  membership  in  a  concept 
class.  Decision  rules  for  six-month  patient  survival  classification  are  extracted  from  the 
dataset  utilizing  genetic  algorithms  for  approximate  reduct  generation. 

Results:  The  RST-based  classifier  performs  comparably  to  other  common  classification 
methods,  while  providing  significant  advantages  in  terms  of  traceability  and  accessibility  of 
the  model.  We  are  in  the  process  of  refining  this  model,  and  hope  to  make  final  decision 
regarding  this  acceptability  for  phase  II  of  our  project. 

Citation:  Eleazar  Gil-Herrera,  Garrick  Aden-Buie,  Ali  Yalcin,  Athanasios  Tsalatsanis,  Laura  E.  Barnes  and 
Benjamin  Djulbegovic,  "Rough  Set  Theory  based  Prognostic  Model  for  Hospice  Referral",  submitted  to 
Journal  of  Artificial  Intelligence  in  Medicine,  Sept.  2012 

Manuscript  7  (appendix  7): 

Title:  Extensions  to  Regret-based  Decision  Curve  Analysis:  An  application  to  hospice  referral 
for  terminal  patients 

In  brief:  Despite  the  well  documented  advantages  of  hospice  care,  most  terminally  ill  patients  do  not 
reap  the  maximum  benefit  from  hospice  services,  with  the  majority  of  them  receiving  hospice  care 
either  prematurely  or  delayed.  Decision  systems  to  improve  the  hospice  referral  process  are  sorely 
needed.  We  present  a  novel  theoretical  framework  that  is  based  on  well-established  methodologies 
of  prognostication  and  decision  analysis  to  assist  with  the  hospice  referral  process  for  terminally  ill 
patients.  We  linked  the  SUPPORT  statistical  model,  widely  regarded  as  one  of  the  most  accurate 
models  for  prognostication  of  terminally  ill  patients,  with  the  recently  developed  regret  based 
decision  curve  analysis  (regret  DCA).  We  extend  the  regret  DCA  methodology  to  consider  harms 
associated  with  the  prognostication  test  as  well  as  harms  and  effects  of  the  management  strategies.  In 
order  to  enable  patients  and  physicians  in  making  these  complex  decisions  in  real-time,  we  developed 
an  easily  accessible  web-based  decision  support  system  available  at  the  point  of  care.  We  present  a 
theoretical  framework  to  facilitate  the  hospice  referral  process.  This  method  uses  our  dual  visual 
analog  scale  (DVAS)  to  elicit  the  patient  preferences  and  values  regarding  the  hospice  referral.  It 
effectively  represents  a  linchpin  between  prognostication  and  elicitation  of  the  patient  values 
regarding  the  choice  they  face  in  terminal  phase  of  their  lives.  This  is  now  integrated  in  our  CDSS-EBM 
and  will  represent  a  cornerstone  of  phase  II  of  our  project. 

Citation:  Tsalatsanis  A,  Barnes  LE,  Hozo  I,  Djulbegovic  B.  Extensions  to  Regret-based  Decision  Curve 
Analysis:  An  application  to  hospice  referral  for  terminal  patients.  BMC  Medical  Informatics  and 
Decision  Making  2011, 11:77  (doi:10.1186/1472-6947-ll-77) 
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Next  Steps 


•  Our  first  step  is  hiring  and  training  research  nurses  to  continue  enrollment  of  participants  in  our  study. 

•  Our  immediate  and  most  important  next  step  is  to  enhance  enrollment  of  patients  in  the  prospective 
phase  of  the  study.  This  requires  tackling  and  coordinating  multiple  logistical,  regulatory  and 
administrative  issues,  which  so  far  we  have  been  successfully  addressing. 

•  Work  very  closely  with  TGH  palliative  team  to  accomplish  the  goals  of  the  study. 

•  We  will  maintain  the  quality  assurance  and  oversight  necessary  for  successful  execution  of  the  study. 

•  We  will  continue  to  test  the  main  module  of  the  EBM-CDSS  computer  program  and  modify  it  as 
needed  (per  feedback  received  from  the  field). 

•  We  will  further  develop  and  complete  testing  the  EB-PMM  (pain  module). 

•  Complete  the  on-going  manuscripts  and  submit  for  publication  in  peer-reviewed  journals. 
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Towards  a  classification  model  to  identify  hospice  candidates  in  terminally  ill  patients 

Eleazar  Gil-Herrera,  Ali  Yalcin,  Athanasios  Tsalatsanis,  Laura  E.  Barnes  and  Benjamin  Djulbegovic 


Abstract —  This  paper  presents  a  Rough  Set  Theory  (RST)  based  classification  model  to  identify  hospice  candidates  within  a 
group  of  terminally  ill  patients.  Hospice  care  considerations  are  particularly  valuable  for  terminally  ill  patients  since  they  enable 
patients  and  their  families  to  initiate  end-of-life  discussions  and  choose  the  most  desired  management  strategy  for  the  remainder 
of  their  lives.  Unlike  traditional  data  mining  methodologies,  our  approach  seeks  to  identify  subgroups  of  patients  possessing 
common  characteristics  that  distinguish  them  from  other  subgroups  in  the  dataset.  Thus,  heterogeneity  in  the  data  set  is 
captured  before  the  classification  model  is  built.  An  alternative  RST  methodology  is  used  to  obtain  the  minimum  set  of  attributes 
that  describe  each  subgroup  existing  in  the  dataset.  As  a  result,  a  collection  of  decision  rules  is  derived  for  classifying  new 
patients  based  on  the  subgroup  to  which  they  belong.  Results  show  improvements  in  the  classification  accuracy  compared  to  a 
traditional  RST  methodology,  in  which  patient  diversity  is  not  considered.  We  envision  our  work  as  a  part  of  a  comprehensive 
decision  support  system  designed  to  assist  terminally  ill  patients  in  making  end-of-life  care  decisions.  Retrospective  data  from 
9105  patients  is  used  to  demonstrate  the  design  and  implementation  details  of  the  classification  model. 

i.  Introduction 


A.  Hospice  referral  criteria 

Hospice  is  designed  to  provide  comfort  and  support  to  terminally  ill  patients  and  their  families. 
According  to  Medicare  regulations,  a  patient  should  be  referred  to  hospice  if  his/her  life  expectancy  is 
approximately  6  months  or  less  [1].  However,  most  patients  are  not  referred  to  hospice  in  a  timely 
manner  [2,  3]  and  therefore  they  do  not  reap  the  well-documented  benefits  of  hospice  services.  A 
premature  hospice  referral  translates  to  a  patient  losing  the  opportunity  to  receive  potentially  effective 
treatment,  which  may  prolong  their  life.  Conversely,  a  late  hospice  referral  may  deprive  patients  and 
their  families  of  enjoying  the  benefits  offered.  Therefore,  accurate  prognostication  of  life  expectancy 
is  of  vital  importance  for  terminal  patients  as  well  as  for  their  families  and  physicians. 

B.  Prognostic  models  for  estimating  survival  of  terminally  ill  patients 

Survival  prognostic  models  range  from  traditional  statistical  and  probabilistic  techniques  [4-10],  to 
models  based  on  artificial  intelligence  such  as  neural  networks  [11,  12],  decision  trees  [13,  14]  and 
rough  set  methods  [15,  16].  The  primary  goal  of  survival  prognostic  models  is  to  provide  accurate 
information  regarding  life  expectancy  and/or  determine  the  association  between  prognostic  factors  and 
survival.  Typically,  the  information  derived  by  prognostic  models  is  presented  in  terms  of  probability 
of  death  within  a  time  period.  Recent  systematic  reviews  [17,  18]  have  highlighted  the  necessity  of 
prediction  models  that  can  be  easily  integrated  into  clinical  practice  and  facilitate  end-of-life  clinical 
decision-making. 

Several  important  issues  demand  particular  consideration  when  developing  clinical  classification 
models:  First,  clinical  data,  representing  patient  records  that  include  symptoms  and  clinical  signs,  are 
not  always  well  defined  and  are  represented  with  vagueness  [19],  Therefore,  it  is  very  difficult  to 
classify  cases  in  which  small  differences  in  the  value  of  an  attribute  may  completely  change  the 
classification  of  a  patient  and,  as  a  result,  the  treatment  decisions  [20],  Second,  clinical  data  may 
present  inconsistencies,  which  means  that  it  is  possible  to  have  more  than  one  patient  with  the  same 
description  but  with  different  outcomes.  Third,  the  results  of  prognostic  models  should  be  readily 
interpretable  to  enable  practical  and  posteriori  inspection  and  interpretation  by  the  treating  physician 
or  an  expert  system  [21],  Finally,  prognostic  models  should  consider  the  heterogeneity  in  clinical  data, 
i.e.  the  existence  of  patient  diversity  presented  in  terms  of  risk  of  disease  and  responsiveness  to 
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treatment  [22,  23],  This  consideration  will  enable  a  prognostic  model  to  identify  possible  subgroups  of 
patients  for  which  certain  covariates  do  not  influence  their  classification.  The  practical  implications  of 
such  considerations  are  associated  with  the  ability  to  customize  the  prognostic  model  for  each 
subgroup  of  patients  (e.g.  expensive  and/or  potentially  harmful  tests  may  be  avoided  for  particular 
subgroups). 

Rough  Set  Theory  (RST)  [24],  a  mathematical  tool  for  representing  and  reasoning  about  vagueness 
and  inconsistency  in  data  sets,  has  been  used  in  a  number  of  applications  dealing  with  modeling 
medical  prognosis  [15,  16,  25-28],  For  example,  Tsumoto  et  al.  [25],  provide  a  framework  to  model 
medical  diagnosis  rules  showing  theoretically  that  the  characteristics  of  medical  reasoning  reflect  the 
concepts  of  approximation  established  in  RST.  Komorowski  et  al.  [26],  show  that  RST  is  useful  to 
extract  medical  diagnosis  rules  to  identify  a  group  of  patients  for  whom  performing  a  test  that  is  costly 
or  invasive  is  redundant  or  superfluous  in  the  prognosis  of  a  particular  medical  condition.  Recently, 
[28]  highlighted  features  of  RST  for  integrating  into  medical  applications.  For  example,  RST  has  the 
ability  to  handle  imprecise  and  uncertain  information  and  provides  a  schematic  approach  for  analyzing 
data  without  initial  assumptions  on  data  distribution. 

In  our  previous  work  [29],  we  proposed  the  use  of  RST  to  predict  the  life  expectancy  of  terminally  ill 
patients  using  a  global  reduction  [30]  methodology  to  identify  the  most  significant  attributes  for 
building  the  classification  model.  However,  we  found  that  the  number  of  attributes  used  in  the  model 
was  barely  reduced  and  therefore  produced  long  decision  rules.  Moreover,  considering  the  number  of 
discretization  categories  associated  with  each  attribute,  the  generated  decision  rules  were  built  to 
describe  each  object  in  the  training  set  and  therefore,  they  were  poorly  suited  for  classifying  new 
cases. 

Here,  we  propose  the  use  of  an  alternative  attribute  reduction  methodology  that  aims  to  identify 
groups  of  patients  that  share  common  characteristics  that  distinguish  them  from  the  rest  of  the  patients. 
As  a  result,  we  obtain  subgroups  of  patients  from  which  different  sets  of  significant  attributes  are 
identified.  The  decision  rules  generated  in  this  manner  contain  fewer  attributes  and  therefore  are  more 
suitable  to  classify  new  patients.  Moreover,  by  studying  each  subgroup,  we  can  reason  about  how  a 
different  rule-set  is  applied  to  a  particular  patient. 

The  rest  of  the  paper  describes  details  of  the  proposed  RST  based  methodology  to  provide  a  classifier 
that  properly  discriminates  patients  into  two  groups:  those  who  survive  at  least  180  days  after 
evaluation  for  hospice  referral  and  those  who  do  not.  ROSETTA  [31]  software  is  used  to  perform  the 
analysis  described  in  the  remainder  of  the  paper. 

II.  Methodology 

A.  Data  Set 

The  dataset  used  in  this  study  consists  of  the  9105  cases  from  the  SUPPORT  (Study  to  Understand 
Prognoses  and  Preferences  for  Outcomes  and  Risks  of  Treatments)  prognostic  model  dataset  [30],  We 
consider  all  variables  used  in  the  SUPPORT  prognostic  model  [3]  as  condition  attributes,  i.e.  the  1 1 
physiologic  variables  along  with  the  diagnosis  groups,  age,  number  of  days  in  the  hospital  before 
entering  the  study,  presence  of  cancer,  and  neurologic  function.  Attributes  names  and  descriptions  are 
listed  in  Table  I.  As  the  decision  attribute,  we  define  a  binary  variable  (Yes/No) 
“deceases_in_6months”  using  the  following  two  attributes  from  the  SUPPORT  prognosis  model 
dataset: 


•  death:  represents  the  event  of  death  at  any  time  up  to  NDI  date  (National  Death  Index  date: 
Dec  31,  1994). 

•  D.time:  number  of  days  of  follow  up 

The  values  of  the  decision  attribute  are  calculated  converting  the  “D.time”  value  in  months  and 
comparing  against  the  attribute  “death”  as  follows: 
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•  If  “D.time”  <  6  months  and  “death”  is  equal  to  1  (the  patient  died  within  6  months)  then 

“deceases_in_6months”  is  “Yes”.  Otherwise,  it  is  implicit  that  a  patient  survived  the  6-month  period; 
hence,  “deceases_in_6months”  is  “No”. 

B.  Rough  Set  Theory  Data  Representation 

Based  on  RST,  the  data  set  is  represented  as: 

T  =  (U,AU{d})  (1) 

TABLE  I.  Condition  Attributes 


Name 

Description 

meanbp 

Mean  arterial  blood  pressure  Day  3 

wblc 

White  blood  cell  count  Day  3 

hrt 

Heart  rate  Day  3 

resp 

Respiratory  rate  Day  3 

temp 

Temperature  (Celsius) 

alb 

Serum  Albumin 

bill 

Bilirubin 

crea 

Serum  Creatinine 

sod 

Sodium 

pafi 

Pa02  /  (.01  *  Fi02) 

ca 

Presence  of  cancer 

age 

Patient’s  age 

hday 

Days  in  hospital  at  study  admit 

dzgroup 

Diagnosis  group 

scoma 

SUPPORT  coma  score  based  on  Glasgow  coma 
scale 

where  T  represents  the  dataset  in  the  form  of  a  table.  Each  row  represents  an  object  and  each  column 
represents  an  attribute.  U  is  a  non-empty  finite  set  of  objects  and  the  set  A  is  a  non-empty  finite  set  of 
attributes  called  the  condition  attributes.  In  our  case,  an  object  designates  a  terminally  ill  patient  and 

an  attribute  a  EA  designates  each  of  the  fifteen  condition  attributes  that  describe  a  patient  (Table  I). 
For  every  attribute,  the  function  a:  U— >Va  makes  a  correspondence  between  an  object  in  U  to  an 
attribute  value  Va  which  is  called  the  value  set  of  a.  The  set  T  incorporates  an  additional  attribute  {d} 
called  the  decision  attribute.  The  system  represented  by  this  scheme  is  called  a  decision  system. 

C.  Development  of  the  Classification  Model 

This  process  typically  involves  numerous  steps,  such  as  data  preprocessing,  discretization,  reduction 
of  attributes,  rule  induction,  classification  and  interpretation  of  the  results.  Details  on  the  data 
preprocessing  and  data  discretization  for  this  data  set  are  described  in  [29].  The  ultimate  goal  of  this 
process  is  to  generate  decision  rules,  which  are  used  to  classify  each  patient  as  surviving  or  not 
surviving  within  the  defined  period  of  time.  A  decision  rule  has  the  form:  if  A  then  B  (A  — >  B),  where 
A  is  called  the  condition  and  B  the  decision  of  the  rule. 

Here,  we  are  focusing  on  an  alternative  method  of  reducing  the  attribute  dimensions  and  identify 
different  subgroups  of  similar  patients  in  the  data  set.  In  [32],  two  types  of  reducts  are  defined: 

1)  Global  Reducts: 

Consists  of  the  minimal  set  of  attributes  that  preserve  the  structure  of  the  entire  data  set  and  can  be 
used  to  generate  decision  rules  and  classify  new  cases.  A  set  B  <=  A  is  called  a  global  reduct  if: 

IND(B)  =  IND(A),  where, 

IND(B )  =  {(ui.Uj)  EU2:  V  ak  E  B,  ak(uO  A  ak(uj)} 

As  an  example,  consider  the  following  global  reduct  obtained  from  the  data  set  containing  12 
condition  attributes: 

G  RED  =  {age,  dzgroup,  scoma,  ca,  meanbp,  wblc,  hrt,  resp,  temp,  bill,  crea,  sod} 
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Using  GRED,  few  patients  will  have  exactly  the  same  attribute-value  combinations  because  the 
number  of  discretization  categories  associated  with  each  attribute  is  high.  Thus,  the  decision  rules 
generated  are  too  specific  to  the  cases  in  the  training  set  and  therefore  may  not  be  able  to  classify  new 
cases  accurately.  Moreover,  the  fact  that  global  reducts  represent  the  entire  data  set  makes  it  difficult 
to  detect  the  presence  of  heterogeneous  groups  in  the  data  meaning  that  the  causes  of  diversity 
between  the  patient  outcomes  will  remain  unknown. 

2)  Object  related  reducts  (ORR): 

Represents  the  minimal  attribute  subsets  that  discern  an  object  u  E  U  from  the  rest  of  objects 
belonging  to  a  different  decision  class.  Mathematically,  an  ORR  Ru  Q  A  is  defined  as: 

V  it;  e  U  :  d(itj)  =£  d (iij )  =>  3  ak  E  Ru :  ak{u{)  =£  ak(uj),  where  iq  =£  Uj  . 

An  ORR  is  the  minimal  and  vital  information  that  is  used  to  partition  the  universe  of  objects  into 
smaller,  homogeneous  subgroups,  where  objects  within  a  subgroup  are  related  by  means  of 
information  described  by  the  ORR.  Decision  rules  generated  by  this  scheme  will  usually  contain  fewer 
attributes  and  are  more  suitable  to  classify  new  cases.  Some  decision  rules  contain  a  different  set  of 
attributes  applicable  for  a  particular  subgroup  of  patients. 

III.  Results 

Based  on  the  decision  rules  generated,  patients  are  classified  as  surviving  or  not  surviving  the  six- 
month  period.  A  standard  voting  algorithm  [30]  is  used  for  this  purpose.  Table  II,  presents  the 
performance  of  two  classification  models  based  on  each  type  of  reduct  generation  described.  The 
performance  of  each  classification  model  is  represented  in  terms  of  sensitivity,  specificity,  Area  under 
the  Receiver  Operating  Characteristic  curve  (AUC)  and  coverage  of  the  model.  A  5-fold  cross 
validation  procedure  was  applied  to  estimate  the  performance  of  each  classification  model,  where,  the 
entire  data  set  is  randomly  divided  into  five  subsets  (folds).  Then,  each  fold  (20%  of  the  data  set)  is 
used  once  as  a  testing  set,  while  the  remaining  folds  (80%)  are  used  for  training.  The  process  is 
repeated  five  times  and  the  results  are  averaged  to  provide  an  estimate  for  the  classifier  performance. 


TABLE  II.  Clasification  results  -  Global  vs.  ORR 


Method 

Sensitivity 

Specificity 

AUC 

Coverage 

Global  Reducts 

73.67% 

44.05% 

61.8% 

86.43% 

ORR 

86.92% 

39.2% 

71.9% 

100% 

Compared  to  the  Global  reduct  approach,  the  ORR  approach  has  enhanced  the  classification 
performance  in  terms  of  AUC  and  sensitivity.  Moreover  the  decision  rules  generated  are  able  to 
classify  all  new  cases. 


IV.  Discussion 

Analyzing  the  information  obtained  from  the  ORR,  we  can  identify  groups  of  patients  for  whom  it  is 
possible  to  evade  costly,  invasive  or  even  unnecessary  tests  required  by  the  prediction  model.  For 
example,  the  following  two  ORRs  generate  rules  independent  of  the  Pafi  score  (associated  with  the 
patient’s  blood  gases),  without  reducing  the  classification  accuracy.  The  importance  of  such  finding 
becomes  apparent  considering  that  in  clinical  practice  Pafi  is  not  collected  routinely  for  patients 
outside  the  Intensive  Care  Unit  (ICU). 

•  ORR  =  {Age,  dzgroup,  meanbp}  generates  the  following  decision  rules: 

o  if  age=  [45,  60)  AND  dzgroup  =  (Lung  Cancer)  AND  meanbp=[60,  70)  then:  Survive  = 
22.86%,  Die  =  77.14%. 

o  if  age=  [45,  60)  AND  dzgroup  =  (CHF)  AND  meanbp=[100,  120)  then:  Survive  =  82.93%, 
Die  =17.07%. 
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o  if  age=  [70,  75)  AND  dzgroup  =  (COPD)  AND  meanbp=[80,100)  then:  Survive  =  84.21%, 
Die  =15.79%. 

•  ORR  =  {Age,  dzgroup,  hrt,  crea}  generates  the  following  decision  rules: 

o  if  age=  [45,  60)  AND  dzgroup  =  (CHF)  AND  hrt=[100,110)  and  crea[1.95,  *]  then:  Survive 
=  83.33%,  Die  =  16.67%. 

o  if  age=  [75,85)  AND  dzgroup  =  (CHF)  AND  hrt=[50,110)  and  crea[0.5,  1.5)  then:  Survive  = 
82.19%,  Die  =  17.81%. 

Consequently,  the  use  of  Pafi  test  in  patients  that  belong  to  one  of  those  groups  defined  by  the  ORR’s 
will  not  improve  the  prognostication  accuracy. 

Our  approach  demonstrates  features  that  make  it  particularly  suitable  for  use  in  clinical  decision¬ 
making.  It  is  a  patient-centric  methodology  which  is  able  to  predict  without  the  use  of  unnecessary, 
expensive  and/or  invasive  procedures  for  certain  subgroups  of  patients.  Consequently,  selection  of 
attributes  upon  which  a  decision  is  to  be  made  is  critical  to  minimizing  healthcare  costs  and 
maximizing  the  quality  of  patient  care.  Finally,  considering  that  more  than  one  ORR  could  discern 
each  patient,  the  information  acquired  offers  several  options  dependent  on  the  attribute  values 
available  for  each  individual  patient. 


V.  Future  Work 

The  number  of  ORR  and  the  decision  rules  generated  depends  on  the  number  of  condition  attributes 
and  its  categories.  For  clinical  datasets,  which  contain  large  numbers  of  condition  attributes,  the 
interpretation  and  analysis  of  the  ORRs  and  their  decision  rules  requires  the  use  of  a  well-defined 
methodology.  Therefore,  we  intend  to  incorporate  this  methodology  into  a  patient-centric  decision 
support  system  to  facilitate  the  hospice  referral  process. 
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ABSTRACT 

Prognostic  models  are  often  used  to  estimate  the  length  of  patient  survival.  The  Cox  proportional 
hazards  model  has  traditionally  been  applied  to  assess  the  accuracy  of  prognostic  models.  However,  it 
may  be  suboptimal  due  to  the  inflexibility  to  model  the  baseline  survival  function  and  when  the 
proportional  hazards  assumption  is  violated.  The  aim  of  this  study  was  to  use  internal  validation  to 
compare  the  predictive  power  of  a  flexible  Royston-Parmar  family  of  survival  functions  with  the  Cox 
proportional  hazards  model.  We  applied  the  Palliative  Performance  Scale  on  a  dataset  of  590  hospice 
patients  at  the  time  of  hospice  admission.  The  retrospective  data  were  obtained  from  the  Lifepath 
Hospice  and  Palliative  Care  center  in  Hillsborough  County,  Florida,  USA.  The  criteria  used  to 
evaluate  and  compare  the  models’  predictive  performance  were  the  explained  variation  statistic  R  , 
scaled  Brier  score,  and  the  discrimination  slope.  The  explained  variation  statistic  demonstrated  that 
overall  the  Royston-Parmar  family  of  survival  functions  provided  a  better  fit  (R  =  0.298;  95%  Cl: 
0.236-0.358)  than  the  Cox  model  (R2  =  0.156;  95%  Cl:  0.111-0.203).  The  scaled  Brier  scores  and 
discrimination  slopes  were  consistently  higher  under  the  Royston-Parmar  model.  Researchers 
involved  in  prognosticating  patient  survival  are  encouraged  to  consider  the  Royston-Parmar  model  as 
an  alternative  to  the  Cox  model. 

Keywords:  Survival  prognostication,  Palliative  Performance  Scale,  Royston-Parmar  survival  models, 
Cox  proportional  hazards,  Internal  validation 
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I. 


INTRODUCTION 


Prognostic  models  are  often  used  to  estimate  the  length  of  patient  survival  and  improvement  in  the 
accuracy  of  prognosis  translates  into  superior  quality  of  patient  care.  Precise  prognosis  of  survival 
using  modeling  techniques  requires  rigorous  methods  for  the  development  and  testing  of  the  accuracy 
of  prognostic  models.  Developing  a  prognostic  model  entails  having  accurate  patient  data  for 
prognosis,  and  selecting  clinically  relevant  candidate  predictors  and  measure(s)  of  model  performance, 
usually  in  the  context  of  a  multivariable  regression  survival  model[l].  This  process  produces  patient 
performance  scores  that  allow  for  classification  of  patients  into  different  risk  groups  [2,3,4], 

In  the  hospice  setting,  accurate  prognostication  of  survival  affords  patients  and  their  families  a 
vital  opportunity  to  attend  to  matters  such  as  planning,  prioritizing,  and  preparing  for  death  [5]. 
Predicting  patient  survival  is  a  complex  decision  making  process  involving  numerous  subjective  and 
numerical  factors  that  have  substantial  variation  which  may  lead  to  poor  prediction  of  life  expectancy. 
Many  physicians  practice  optimism  or  avoidance,  thus  overestimating  survival  at  times  by  a  factor  of 
five[6].  Implementing  appropriate  statistical  methodologies  translates  into  improved  accuracy  of 
prognosis  and  superior  quality  of  care.  Predictions  based  on  appropriate  statistical  modeling  have  been 
shown  to  be  superior  to  physicians’  prognostication  [4,7]. 

The  Cox  proportional  hazards  (CPH)  model[8]  is  the  most  commonly-used  survival  prediction 
model[4,9].  In  the  hospice  and  palliative  settings,  demographic  and  clinical  covariates  are  often 
included  in  CPH  to  predict  patient  survival  [10,1 1].  The  appeal  of  the  model  is  its  analytic  simplicity 
and  that  the  baseline  survival  function  does  not  need  to  be  defined  apriori  —  it  is  absorbed  when  the 
likelihood  function  is  maximized  (note  that  “baseline”  refers  to  zero  values  of  the  covariates,  not  to 
time  equal  to  zero).  It  is  possible  to  estimate  the  baseline  survival  function  for  the  CPH  model 
conditional  on  the  estimated  regression  coefficients.  However,  this  is  highly  rigid  as  the  smoothing  of 
the  underlying  function  depends  on  the  proportional  hazards  assumption,  which  may  not  be  supported 
by  the  data  and  is  often  overlooked  by  the  investigators  [9],  Essentially,  the  CPH  model  was  designed 

to  measure  the  effects  of  covariates  on  the  changing  hazard  function  and  not  to  model  patient  survival. 
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A  flexible  family  of  functions  which  allows  for  parametrically  modeling  the  baseline  survival 
function  is  more  appropriate,  especially  if  the  proportional  hazards  assumption  is  violated  in  the 
CPH[12],  The  baseline  survival  has  for  the  most  part  been  ignored  because  it  is  left  undefined  in  the 
CPH  model. 

In  this  manuscript  we  compare  CPH  with  an  alternative  method  of  estimating  survival  in  the  form 
of  the  class  of  flexible  Royston-Parmar  (RP)  parametric  functions[12],  We  use  the  Palliative 
Performance  Scale  (PPS)[13]  from  a  cohort  of  hospice  patients.  Results  from  systematic  reviews  have 
shown  that  the  patient  PPS  score  is  an  accurate  measure  of  patient  survival  in  the  palliative  setting 
[7,1 1].  Furthermore,  PPS  and  CPH  model  have  been  used  to  construct  meaningful  hospice  patient 
survival  estimates  in  the  form  of  a  life  expectancy  table  and  survival  nomogram  [14]. 

In  addition  to  PPS,  other  risk  factors  such  as  age,  cancer  status  and  gender  have  been  reported  to 
be  significant  predictors  of  palliative  patient  survival  in  several  studies  [1 1,14].  In  our  study  we  did 
not  adjust  for  other  risk  factors  because  though  they  may  be  significant  predictors  of  survival  for  the 
cohort  of  patients  in  our  dataset,  they  may  not  be  in  other  palliative  settings.  Our  goal  was  to 
demonstrate  that  the  RP  family  of  parametric  functions  allowed  for  a  direct  and  flexible  modeling  of 
the  baseline  survival  and  that  it  might  be  formulated  so  that  the  impact  of  the  proportional  hazard 
assumption  is  minimized.  We  determined  if  the  overall  performance  and  discriminatory  ability  of  RP 
family  of  parametric  functions  is  superior  to  CPH  in  the  sample  by  using  models  that  were  derived  and 
tested  on  the  whole  dataset  (naive  internal  validation)  and  using  (internal)  cross-validation.  CPH  has 
been  widely  used  to  validate  prognosticating  scales  for  hospice  patient  survival  [15,16,17].  It  is 
important  to  note  that  the  RP  parametric  functions  have  not  been  applied  to  prognostic  models  in  the 
hospice  and  palliative  settings.  It  is  also  important  to  note  that  we  did  not  perform  external  validation, 
which  entailed  using  a  different  data  set  than  the  one  used  to  create  the  model[3].  In  the  next  section 
we  briefly  discuss  PPS,  introduce  the  statistical  models  and  measures  of  model  performance. 

II.  METHODS 
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Study  sample  and  palliative  performance 

The  patient  data  were  obtained  from  the  Lifepath  Hospice  and  Palliative  Care  Center,  licensed  since 
1983  to  serve  in  Hillsborough  County,  Florida.  Hospice  care  focuses  on  pain  control  and  symptom 
management.  To  avoid  selection  bias,  we  retrospectively  and  sequentially  extracted  data  for  590 
patients  who,  as  of  January  2009  were  deceased.  This  study  was  a  retrospective  review  of  the  deceased 
patients’  medical  records.  Only  data  pertaining  to  outcomes  were  collected;  personal  information  was 
not  collected  and  the  data  were  de-identified  prior  to  analysis.  Since  we  did  not  collect  any 
information  that  can  identify  deceased  patients  or  their  family  members,  under  HIPPA  rules  and 
regulations  (45  CFR  164.512)  the  requirement  for  consent  does  not  apply.  The  study  and  consent 
procedures  were  approved  by  the  University  of  South  Florida  Institutional  Review  Board  prior  to 
study  initiation.  Two  research  assistants  extracted  all  data  necessary  to  populate  the  model  variables 
and  two  faculty  members  randomly  checked  25%  of  the  data  for  accuracy.  The  models  were  tested 
against  observed  survival  duration. 

The  Palliative  Performance  Scale  (PPS)  was  developed  and  reported  by  Anderson  et  al.  [13]  as 
a  measure  of  palliative  patients’  functional  status.  The  scale  has  11  possible  mutually  exclusive  levels, 
which  are  based  on  five  domains:  six  levels  of  ambulation,  six  levels  of  activity  and  evidence  of 
disease,  five  levels  of  self-care,  five  levels  of  food  intake  and  four  levels  of  consciousness.  The  scale 
ranges  from  PPS  of  0%  (deceased  patient)  to  PPS  of  100%  (ambulatory  and  healthy  patient). 
Numerous  studies  have  studied  its  prognostic  accuracy  of  survival  in  a  variety  of  settings  and  found  it 
provides  meaningful  estimates  of  patient  survival  [10,14,15,18,19,20,21,22,23].  PPS  has  been  found  to 
be  both  valid  and  reliable  [24], 

Model  selection  and  validation 

Validating  a  prognostic  model  is  generally  accepted  to  mean  that  given  a  patient  population  it  works  in 
a  data  set  other  than  the  one  it  is  applied  to[2,25].  In  other  words,  the  model  needs  to  be  tested  using  a 
different  data  set  than  the  one  used  to  create  the  model[3].  It  is  also  generally  accepted  that  the 


Annual  progress  report  page  21 


validation  process  should  follow  guidelines  and  that  un- validated  prognostic  models  should  not  be 
applied  in  clinical  practice  [3,4,26].  When  validating  a  prognostic  survival  model  in  the  regression 
framework,  most  attention  has  been  on  the  value  of  the  prognostic  index  based  on  covariates,  while 
the  role  of  the  baseline  survival  function  has  been  largely  ignored.  The  role  of  the  baseline  survival  is 
significant  as  it  quantifies  the  absolute  patient  survival  probabilities  over  time.  For  a  vector  of 
covariates  x  and  parameter  vector  p,  the  survival  function  S(t;  x)  at  time  t  for  the  CPH  model  is 
commonly  expressed  as  S(t;x)  =  [S,0(f)]exp(*'ff),  where  So  (t)  is  the  baseline  survival  function,  i.e. 

survival  function  when  all  the  covariates  x  are  equal  to  zero.  In  the  CPH  framework,  the  estimation  of 
the  prognostic  index  xp  does  not  require  the  formulation  of  the  baseline  cumulative  survival  function 
So  (t),  which  itself  can  be  estimated  conditional  on  the  covariate  estimates.  The  two  popular  methods 
for  estimating  baseline  survival  So  (t)  are  the  Breslow  and  Kalbfleisch-Prentice  methods  [27].  Both 
give  similar  results  in  practice,  but  can  lead  to  “choppy”  estimates  of  the  baseline  function  and  are 
dependent  on  the  proportional  hazards  assumption. 

When  the  goal  of  a  survival  analysis  is  to  estimate  hazard  ratios  (the  effect  of  covariates  on  the 
changing  hazard  function),  the  baseline  function  is  of  no  consequence.  The  CPH  is  appropriate  as  the 
baseline  function  gets  absorbed  when  coefficient  Ps  are  estimates  by  the  method  of  partial  log 
likelihood.  However,  when  the  goal  is  to  prognosticate  patient  survival,  there  is  a  need  for  more 
flexibility  in  modeling  the  baseline  survival. 

An  alternative  to  the  CPH  is  the  RP  family  of  models  that  resembles  the  generalized  linear 
models  and  can  be  viewed  as  a  parametric  extension  Cox  proportional  hazard  models  [12].  The 
models  are  framed  to  rely  on  the  transformation  g(.),  such  that  g(S(t;x))  =  g(S0(t ))  +  xfi .  The 

transformation  g(.)  can  be  either  from  the  proportional  hazard,  proportional  odds,  Aranda-Ordaz  or 
probit  families  [12].  We  did  not  consider  the  Aranda-Ordaz  family  in  this  study  due  to  possible 
interpretational  difficulties  [12].  Under  the  proportional  hazard  link  function,  the  hazard  ratio 
estimates  are  nearly  identical  to  those  estimated  under  CPH.  The  attractive  feature  of  the  RP  baseline 
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survival  function  is  that  its  shape  is  preserved,  but  the  location  of  the  baseline  distribution  function  can 
vary,  which  allows  for  flexible  model  recalibration.  Also,  the  estimate  g(S0(t))  is  implemented  on 
log-time  scale.  It  is  generally  gently  curved  and  smooth,  making  survival  estimates  more  accurate. 

In  the  RP  framework,  if  the  proportional  hazard  assumption  is  violated,  the  probit-link  function 
g(s)  =  -  O' 1  (s)  can  be  applied,  where  O' 1  (.)  is  the  inverse  standard  normal  distribution  function.  The 
baseline  survival  function  S0  (t)  is  approximated  and  smoothed  by  a  restricted  cubic  spline  function 
with  m  interior  knots.  Splines  are  piecewise  polynomials  that  ensure  the  overall  curve  is  smooth  (see 
Royston  and  Parmar  [12]  for  details).  Spline-based  survival  models  such  as  RP  have  been  empirically 
shown  to  be  superior  when  the  proportional  hazard  assumption  is  violated[28].  The  optimal  number 
of  knots  and  the  comparison  among  different  RP  models  can  be  found  using  the  minimum 
combination  of  Akaike  Information  Criterion  (AIC)  Bayes  Information  Criterion  (BIC)  and  explained 
variation  statistic  R2[29,30],  The  AIC  is  defined  in  the  usual  manner  as  -  2Log(likelihood)  +  2(No.  of 
model  parameters),  while  BIC  equals  -  2Log(likelihood)  +  (No.  of  model  parameters)  *Log(n).  In 
survival  analysis  n  is  interpreted  as  the  number  of  events  rather  than  the  number  of  patients. 

We  compared  RP  and  CPH  by  performing  internal  validation  (assessing  validity  in  the 
population  where  the  development  data  originated  from)  on  the  whole  data  set  (naive)  and  using  split- 
sample  cross-validation.  We  performed  10-fold  cross-validation  by  splitting  the  data  into  development 
and  validation  sets  and  repeating  the  process  20  times.  The  methods  can  be  readily  implemented  in 
Stata  [31,32]  statistical  software  using  the  stpm[ 29]  and  stpm2\ 33]  commands,  or  in  open  source 
statistical  software  R  as  flexsurv  package[34]. 

Assessment  of  model  performance 

Model  performance  is  the  ability  of  the  estimated  risk  score  to  predict  survival  and  is  assessed  using 
the  measures  of  explained  variation,  calibration,  and  discrimination.  Calibration  refers  to  how  closely 
the  predicted  survival  at  a  pre-specified  time  agrees  with  the  observed  survival.  For  cross-validation, 
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we  compared  the  average  fitted  probabilities  of  survival  under  RP  and  CPH  for  the  first  1 5  days  to 
observed  probabilities  estimated  non-parametrically  using  Kaplan-Meier  curves[35]. 

The  Brier  score  is  a  quadratic  scoring  rule  that  calculates  the  differences  between  the  actual 
outcomes  and  predicted  probabilities [3 6].  Given  the  predicted  probability  of  survival  p,  at  time  t  for 
patient  i,  and  Y;  binary  (0-1,  dead-alive)  variable,  the  Brier  score  is  defined  as  £i(Yj  (1  —  pj)2  + 

(1  —  Yj)  Pi2).  A  Brier  score  of  0  indicates  a  perfect  model,  while  0.25  indicates  a  non-informative 
model  (the  value  achieved  when  issuing  a  predicted  probability  of  50%  to  each  patient).  The  Brier 
score  may  be  scaled  by  its  maximum  Briermax  =  (1  -  mean(p,))  mean(pi)  to  obtain  Brierscaied  = 

/  Brier  \ 

1 1  —  — - )  100%.  The  scaled  Brier  scores  range  from  0%  to  100%  and  have  interpretation  similar 

to  the  Pearson  correlation  coefficient[37]. 

For  a  particular  risk  score,  discrimination  is  the  ability  to  differentiate  between  the  patients  who 
died  versus  those  who  survived.  The  Kaplan-Meier  plot  of  survival  for  patients  in  different  risk 
groups  can  be  used  to  test  for  separation,  indicating  that  the  different  risk  groups  are  well  defined[38]. 
For  a  statistical  model,  the  global  measure  of  the  model’s  discriminatory  power  is  the  explained 
variation  statistic  R  ,  which  measures  the  variation  explained  by  the  fitted  model[39].  Higher  values 
of  R2  indicate  greater  discrimination.  In  this  study  we  implement  R2  for  survival  models,  as  described 
by  Royston  and  Sauerbrei[40], 

The  discrimination  or  Yates  slope  is  a  measure  of  how  well  the  subjects  with  and  without  the 
outcome  are  separated.  It  is  defined  as  the  absolute  difference  in  mean  predictions  of  survival 
(mean[pi])  between  those  who  died  and  those  who  survived  at  time  t[2].  The  scaled  Brier  scores  and 
discrimination  slopes  were  calculated  separately  for  the  (naive)  model  using  the  whole  dataset  and  the 
model  derived  using  cross-validation.for  t  =  1,2...  100  days.  Higher  scaled  Brier  scores  and 
discrimination  slopes  represent  better  model  performance. 

All  statistical  calculation  were  performed  using  Stata  version  1 1.2[31,32], 

III.  RESULTS 
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Description  of  the  data  source 

The  patient  characteristics  of  the  retrospective  cohort  are  summarized  in  Table  1.  The  cohort 
consisted  of  293  males  (49.7%)  and  295  females  (50.0%),  and  2  (0.3%)  with  unknown  gender.  The 
data  were  collected  starting  from  patients’  entry  into  hospice  care  until  death  for  all  590  patients.  The 
mean,  median  and  range  of  survival  times  for  the  patients  by  PPS  at  admission,  age,  gender,  cancer 
status,  and  diagnosis  category  are  given  in  Table  2.  The  table  shows  that  the  median  survival  was 
fairly  evenly  distributed  across  age  groups  and  gender,  but  unevenly  across  the  cancer  status  and 
initial  diagnosis  category.  All  patients  were  assigned  PPS  at  the  time  of  admission  to  hospice  care. 
Since  PPS  score  of  0%  means  that  the  patient  is  dead,  the  data  were  transformed  so  that  the  PPS  score 
of  10%  was  set  as  the  baseline.  There  were  only  15  total  observations  for  PPS  =  60%,  70%,  80%,  so 
they  were  combined  with  PPS  =  50%  to  obtain  meaningful  survival  estimates.  Fourteen  patients  had 
missing  values  for  PPS. 

The  time  of  admission  was  the  starting  point  for  survival  time.  The  Kaplan-Meier  curves 
stratified  by  initial  PPS  level  are  shown  in  Figure  1.  The  curves  show  good  separation  indicating  that 
the  different  risk  groups  are  well  defined.  The  log-rank  test  for  equality  of  survival  curves  was  highly 
significant  at  P  =  0.0001.  The  global  test  based  on  Schoenfeld  residuals  showed  that  the  proportional 
hazard  assumption  was  violated  for  PPS  (P-value<0.001),  which  can  also  be  seen  from  the  un-parallel 
log-plot  of  survival  curves  (Figure  2). 

Table  3  lists  AIC,  BIC  and  R2  values  for  5  knots  under  the  proportional  hazard,  proportional 
odds  and  probit  RP  families;  the  minimum  combination  in  each  underlined.  The  number  of  optimal 
knots  was  found  to  be  m=l  under  the  probit  model.  The  improvement  in  fit  with  the  probit  model  can 
be  seen  from  the  parallel  survival  curves  of  log-probit  against  log  time  (Figure  3). 

R2  was  higher  in  the  RP  model  (R2  =  0.298;  95%  Cl:  0.236-0.358)  than  the  Cox  model  (R2  = 

0.156;  95%  Cl:  0.1 1 1-0.203),  indicating  that  the  RP  model  explained  significantly  more  variation  than 
CPH.  To  illustrate  the  differences  for  the  baseline  function,  Figure  4  shows  plots  of  the  CPH  and  RP 
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baseline  survival  functions.  The  CPH  baseline  survival  is  “choppy”  to  approximately  day  12,  while  the 
RP  is  smooth.  The  two  baseline  functions  converged  at  around  day  12. 

Cross-validation  showed  that  the  relation  between  the  two  predicted  survival  estimates  is 
approximately  linear,  with  RP  model  consistently  estimating  a  higher  probability,  which  is  particularly 
evident  for  higher  scores  of  PPS  corresponding  to  longer  survival  times  (Figure  5).  Overall,  the 
predicted  probabilities  under  RP  tended  to  be  closer  to  the  Kaplan-Meier  estimates  than  CPH.  The  plot 
of  the  consistently  positive  differences  between  RP  and  CPH  scaled  Brier  scores  and  discrimination 
slopes  showed  that  the  RP  model  discriminated  better  across  patient  survival  times  for  both  the  full 
(naive)  (Figure  6a)  and  cross- validated  models  (Figure  6b).  This  suggested  that  the  higher  value  of  R2 
under  RP  was  not  due  to  over-fitting. 

IV.  DISCUSSION 

The  results  from  our  study  show  that  RP  family  of  models  predicts  survival  more  accurately  than  CPH 
through  its  flexible  modeling  of  the  baseline  survival  function.  Using  the  RP  flexible  baseline  function 
modeling  would  allow  for  more  precise  calibration  in  the  prognostication  phase  than  CPH.  As  Figure 
5  illustrates,  the  predicted  RP  survival  probabilities  are  consistently  higher  for  higher  values  of  PPS, 
and  closer  to  the  Kaplan-Meier  estimates  of  survival.  We  suspect  that  both  the  robust  modeling  of 
baseline  survival  and  overall  model  fit  provide  for  better  survival  estimation. 

There  are  limitations  to  our  study,  the  primary  one  being  the  use  of  retrospective  data.  The  RP 
family  of  parametric  functions  needs  to  be  applied  prospectively  to  assess  accuracy  of  prognostic 
models  through  external  validation.  Furthermore,  the  dataset  was  limited  to  the  hospice  setting  with  no 
censored  observations  with  majority  of  patients  having  a  very  short  follow-up  time.  For  future 
studies,  application  of  the  proposed  methodology  should  account  for  these  limitations,  and 
comparisons  with  parametric  prognostic  survival  models  should  be  explored. 
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The  flexible  models  discussed  in  this  paper  could  greatly  improve  the  ability  of  researchers  to 
accurately  predict  survival.  An  advantage  of  RP  is  that  it  can  be  used  to  validate  published  models  for 
which  the  original  individual  patient  data  are  unavailable.  If  the  scale  used  (hazard,  probit  or  odds), 
the  knot  positions,  and  the  estimates  of  prognostic  indices  are  known,  then  it  would  be  possible  to  use 
RP.  In  the  case  of  CPH  this  is  not  possible,  since  the  baseline  function  would  not  be  available. 
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Figure  Legends 

Figure  1.  Kaplan-Meier  survival  curves  by  initial  PPS. 

Figure  2.  Test  of  proportional  hazards  assumption  under  CPH  for  initial  PPS. 

Figure  3.  Test  of  the  probit  assumption  under  probit  RP  for  initial  PPS. 

Figure  4.  Baseline  survival  functions  under  CPH  and  RP  models. 

Figure  5.  Predicted  probabilities  under  RP,  and  Cox  models,  plotted  against  the  Kaplan-Meier 
estimates  of  survival  in  the  validation  set. 

Figure  6.  Difference  between  Scaled  Brier  scores  (6a)  and  discrimination  slopes  (6b)  under  RP  and 
Cox  full  (naive)  and  cross-validated  models,  as  a  function  of  patient  survival  times.  Both  are 
consistently  higher  for  RP  indicating  better  discrimination  under  both  naive  and  cross-validated 
models. 
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Tablel.  Patient  Characteristics 

Variable 

Result 

Total  no.  of  patients 

590(100%) 

Age  at  Treatment 

<45 

37(6.3%) 

45-64 

187(31.7%) 

65-74 

110(18.6%) 

75-84 

129(21.9%) 

85+ 

127(21.5%) 

Gender 

Male 

293(49.7%) 

Female 

295(50%) 

Unknown 

2(0.3%) 

No.  of  patients  with  cancer/noncancer 

Noncancer 

363(61.5%) 

Cancer 

227(38.5%) 

Diagnosis  category  for  cancer 

Brain 

10(1.7%) 

Gastrointestinal 

35(5.9%) 

Genital-female 

12(2%) 

Genital-male 

12(2%) 

Head  and  neck 

8(1.4%) 

Hematopoietic 

10(1.7%) 

Pancreas 

25(4.2%) 

Respiratory 

49(8.3%) 

Skin 

2(0.3%) 

Urinary 

4(0.6%) 

Other 

61  (10.3%) 

Diagnosis  category  for  noncancer 

AIDS 

12(2%) 

Cardiovascular 

74(12.5%) 

Neurological 

69(11.7%) 

Respiratory 

37(6.3%) 

Other 

118(20%) 

Table  1.  Patient  Characteristics 
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Table2.  Survival  time  by  age,  gender,  diagnosis  and  IntMal  PPS 


Survival  Times  (in  Days) 

Variable 

Mean  (95%  Cl) 

Median  (95%  Cl) 

Range 

No.  of  Patients  (•/•) 

Total  no.  of  patients 

Overall 

14(12,17) 

6(5,6) 

1-371 

590 

Age  at  Treatment 

<45 

15(8,22) 

8(4,12) 

1-95 

37(6.3%) 

45-64 

14(11,17) 

7(5,9) 

1-114 

187(31.7%) 

65-74 

14(8,20) 

5(4,6) 

1-271 

110(18.6%) 

75-84 

14(8,20) 

6(5,7) 

1-371 

129(21.9%) 

85+ 

15(9,21) 

5(4,6) 

1-313 

127(21.5%) 

Gender 

Male 

14(10,18) 

6(5,7) 

1-371 

293(49.7%) 

Female 

15(11,19) 

6(5,7) 

1-271 

295(50%) 

No.  of  patients  with  cancer/noncancer 

Noncancer 

12(8, 16) 

5(4,6) 

1-371 

363(61.5%) 

Cancer 

17(14,20) 

9(7,11) 

1-113 

227(38.5%) 

Diagnosis  category  for  cancer 

Brain 

27(16,39) 

28(14,42) 

3-55 

10(1.7%) 

Gastrointestinal 

21(14,29) 

11(5,17) 

1-82 

35(5.9%) 

Genital-female 

15(6, 24) 

8(1,15) 

2-55 

12(2%) 

Genital-male 

26(7, 45) 

13(4,22) 

1-100 

12(2%) 

Head  and  neck 

10(2,18) 

5  0,9) 

1-36 

8(1.4%) 

Hematopoietic 

4(2,6) 

3(1,5) 

1-10 

10(1.7%) 

Pancreas 

18(7, 29) 

7(3,11) 

1-113 

25(4.2%) 

Respiratory 

15(10,20) 

10(7, 13) 

1-71 

49(8.3%) 

Skin 

11 

11 

11-11 

2(0.3%) 

Urinary 

25(1-58) 

9(1-39) 

4-76 

4  (0.6%) 

Other 

17(12,22) 

9(5, 12_ 

1-103 

61  (10.3%) 

Diagnosis  category  for  noncancer 

AIDS 

18(3, 33) 

8(1,15) 

1-85 

12(2%) 

Cardiovascular 

14(5,23) 

5(3,7) 

1-271 

74(12.5%) 

Neurological 

8(5,11) 

5(4,6) 

1-77 

69(11.7%) 

Respiratory 

25(1,49) 

30,5) 

1-371 

37(6.3%) 

Other 

11(1,15) 

5(4,6) 

1-174 

118(20%) 

Initial  PPS  Score 

PPS  10% 

5(3,7) 

3(2,4) 

1-77 

188(32.6%) 

PPS  20% 

16(8, 24) 

5(4,6) 

1-371 

125(21.7%) 

PPS  30% 

15(11,19) 

7(5,9) 

1-140 

123(21.4%) 

PPS  40% 

24(18,30) 

14(11,17) 

1-147 

96(16.7%) 

PPS  50-80% 

28(21,35) 

18(9, 27) 

1-76 

44(7.6%) 

Table  2.  Survival  Time  by  Age,  Gender,  Diagnosis  and  Initial  PPS. 
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Table  3.  Choice  of  scale 


No.  of  knots  m 

PH 

AIC,  BIC,  R2 

PO 

AIC,  BIC,  R2 

Probit 

AIC,  BIC,  R2 

0 

2033,2042,0.156 

1887,1896,0.321 

1872,1881,0.295 

1 

1889,1902,0.178 

1883,1896,0.322 

1858,1871.0.298 

2 

1871.1888,0.170 

1870,1887,0.312 

1857,1874,0.296 

3 

1870,1892,0.172 

1870,1892,0.311 

1858,1880,0.297 

4 

1865.1892.0.171 

1865.1891.0.310 

1855,1881,0.296 

5 

1866,1896,0.171 

1865,1896.0.309 

1856,1886,0.296 

Table  3.  Choice  of  scale  under  RP  proportional  hazard,  proportional  odds  and  probit  models. 
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Figure  4 
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Figure  5 
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Purpose:  To  conduct  a  systematic  review  and  meta-analysis  of  the  natural  history  of  patients  with 
confirmed  diagnosis  of  lung  cancer  without  active  treatment. 


Methods:  Relevant  studies  were  identified  by  search  of  MEDLINE  (PubMed)  and  CENTRAL 
electronic  databases  and  abstract  proceedings  up  to  June  2011.  All  prospective  or  retrospective 
studies  assessing  prognosis  of  lung  cancer  patients  without  treatment  were  eligible  for  inclusion. 
Data  on  mortality  was  extracted  from  all  included  studies  and  pooled  proportion  of  mortality  was 
calculated  as  a  back-transform  of  the  weighted  mean  of  the  transformed  proportions,  using  the 
random-effects  model. 


Results:  Seven  cohort  studies  (4,418  patients)  and  15  randomized  controlled  trials  (1,031  patients) 
were  included  in  the  meta-analysis.  All  studies  assessed  mortality  without  treatment  in  patients 
with  non-small  cell  lung  cancer  (NSCLC).  The  pooled  proportion  of  mortality  without  treatment  in 
cohort  studies  was  0.97  (95%  Cl:  0.96  to  0.99)  and  0.96  in  randomized  controlled  trials  (95%  Cl: 
0.94  to  0.98)  over  median  study  periods  of  8  and  3  years,  respectively.  The  pooled  proportion  of 
mortality  was  0.97  (95%  Cl  0.96  to  0.98)  when  data  from  cohort  and  randomized  controlled  trials 
were  combined.  Test  of  interaction  showed  a  statistically  non-significant  difference  between 
subgroups  of  cohort  and  randomized  controlled  trials.  Overall  the  studies  were  of  moderate 
methodological  quality. 


Conclusion:  Systematic  evaluation  of  evidence  on  prognosis  of  NSCLC  without  treatment  shows 
that  mortality  is  very  high.  Although  limited  by  study  design,  these  findings  provide  the  basis  for 
future  trials  to  determine  optimal  expected  improvement  in  mortality  with  innovative  treatments. 


INTRODUCTION 


Cancer  is  a  major  public  health  concern  globally.  It  is  the  most  frequent  cause  of  death  in 
economically  developed  countries.1  Among  all  cancers,  lung  cancer  is  the  leading  cause  of  cancer 
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deaths  worldwide. 2  In  the  United  States,  approximately  221,130  new  cases  of  lung  cancer  (14%  of 
all  cancer  diagnoses)  are  expected  in  201 1  out  of  which  156,940  deaths  (27%  of  cancer  deaths)  are 
estimated  due  to  lung  cancer.  Given  the  incurative  nature  of  lung  cancer,  it  is  considered  a  terminal 
illness  with  a  5-year  survival  rate  of  approximately  16%  .3 

Patients  diagnosed  with  terminal  illness  such  as  lung  cancer  confront  several  decisions  related 
to  management  of  the  disease.  Opting  for  treatment  (e.g.  chemotherapy,  radiotherapy,  or  surgery) 
instead  of  palliation  or  vice  versa  is  one  such  critical  decision.  Depending  on  the  stage  of  the  disease, 
potential  benefits  of  anticancer  therapy  intended  to  palliate  specific  tumor-related  symptoms  may  be  at 
the  expense  of  treatment-related  harms  and  the  inconvenience  associated  with  undergoing  treatment. 
Other  times,  palliative  care  (e.g.  pain  medications  or  low  dose  radiotherapy)4  rather  than  anticancer 
therapy  may  be  preferable.  Informed  decision  related  to  management  of  a  terminal  disease  thus 
requires  accurate  prognosis  of  the  disease  with  or  without  treatment. 

Briefly,  prognosis  refers  to  the  likelihood  of  an  individual  developing  a  particular  health 
outcome  over  a  given  period  of  time,  based  on  the  individual’s  clinical  and  non-clinical  profile.5 
Accurate  assessment  of  prognosis  is  key  to  informed  decision  making.  For  example,  if  a  patient  is 
diagnosed  with  a  terminal  illness  such  as  lung  cancer,  a  prognostic  question  of  critical  concern  to  the 
patient,  family,  and  the  physician  is  how  long  the  patient  is  expected  to  live.  Other  important 
outcomes  may  include  disease  progression,  health-related  quality  of  life,  and  treatment-related  harms. 
Reliable  prognostication  of  life  expectancy  can  prevent  subjecting  patients  to  costly  and  unnecessary 
treatment  for  an  unduly  long  period  before  transitioning  to  hospice  care.6  This  in  turn  can  help  patients 
and  their  families  prepare  for  the  impending  events  and  plan  for  the  patient’s  remaining  lifespan.7 
Accurate  prognostic  information  can  also  help  physicians  decide  on  choice  of  curative  versus 
palliative  treatments.  For  instance,  if  evidence  shows  no  effect  of  curative  treatment  on  disease 
progression,  significant  treatment-related  harms  can  be  avoided  in  favor  of  palliative  treatments.7 
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Accurate  disease  prognosis  thus  underpins  all  management  decisions  related  to  the  disease  including 
choice  of  treatment,  planning  of  supportive  care,  as  well  as  allocation  of  resources. 

Despite  the  significance  of  disease  prognosis  in  clinical  decision-making,  systematic 
assessment  of  prognosis  in  patients  with  lung  cancer  without  treatment  has  not  been  performed.  We 
are  aware  of  only  one  narrative  review  on  the  subject.8  Accordingly,  this  systematic  review  was 
undertaken  to  assess  the  natural  history  of  patients  with  confirmed  diagnosis  of  lung  cancer  without 
active  treatment.  Specifically,  our  aim  was  to  estimate  overall  survival  (natural  history)  in  lung  cancer 
when  no  anticancer  therapy  is  provided. 

METHODS 

This  systematic  review  was  conducted  as  per  the  methods  elaborated  in  a  protocol  that  was  developed 
a  priori.  An  ideal  study  design  to  assess  natural  history  of  a  terminal  disease  such  as  lung  cancer  is  a 
cohort  study.  Specifically,  an  inception  cohort  whereby  a  well-defined  group  of  patients  at  the  same 
disease  stage  is  assembled  at  first  diagnosis  and  followed  for  a  defined  period  of  time.9'11  However, 
given  the  availability  of  treatments  for  lung  cancer  in  recent  years,  it  would  be  unethical  and 
logistically  challenging  to  conduct  such  a  study.  An  alternative  approach  is  to  assess  prognosis  from 
retrospective  lung  cancer  registries,  case  series  or  from  the  control  arm  of  individual  RCTs  that 
compare  active  treatment  with  either  no  treatment  or  placebo  or  best  supportive  care.''  Thus,  in  this 
review,  any  retrospective  or  prospective  cohort  study  assessing  prognosis  in  lung  cancer  without 
treatment  and  any  RCT  assessing  the  role  of  treatment  versus  no  treatment,  were  eligible  for  inclusion. 
A  study  was  eligible  for  inclusion  irrespective  of  language  or  publication  type. 

Search  Strategy 

We  conducted  a  systematic  search  of  PubMed  and  Cochrane  library  electronic  databases, 
proceedings  of  major  scientific  meetings,  and  bibliographies  of  eligible  studies  to  identify  all  relevant 
studies.  To  retrieve  lung  cancer  prognosis  studies  in  PubMed,  we  employed  search  strategies 
suggested  by  Wilczynski1'1  that  optimizes  search  sensitivity  and  specificity.  Search  details  used 
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included:  ("lung  neoplasms" [MeSH  Terms]  AND  "prognosis"[All  Fields]  AND  "cohort"[All  Fields] 
AND  ("mortality" [Subheading]  OR  "natural  course"[All  Fields]  OR  "mortality" [All  Fields]  OR 
"survival" [All  Fields]  OR  "survival"[MeSH  Terms]). 

To  retrieve  RCTs  in  PubMed,  we  employed  strategies  suggested  by  Haynes14  with  the 
following  search  details:  ("lung  neoplasms" [MeSH  Terms]  AND  ("randomized  controlled 
trial" [Publication  Type])  AND  ("palliative  care"[All  Fields]  OR  "hospice  care"[All  Fields]  OR 
"supportive  care"[All  Fields]  OR  "best  supportive  care"[All  Fields]  OR  "placebo"[All  Fields]  OR 
"symptomatic  treatment" [All  Fields]  OR  "no  chemotherapy" [All  Fields]  OR  "no  treatment" [All 
Fields]). 

In  the  Cochrane  library,  we  utilized  a  free  text  search  using  the  term  “Lung  cancer”  to  identify 
RCTs  focusing  on  lung  cancer.  We  manually  searched  abstracts  of  the  American  Society  of  Clinical 
Oncology  (ASCO)  and  American  Society  of  Hematology  (ASH)  meetings  and  utilized  the 
snowballing  procedure  to  identify  other  relevant  studies.  Studies  published  until  June  2011  were 
included.  No  restrictions  were  made  regarding  the  language  of  the  publication. 

Inclusion  and  Exclusion  Criteria 

A  prospective  or  retrospective  cohort  study  assessing  overall  survival  as  an  outcome  in  lung 
cancer  patients  without  treatment  was  eligible  for  inclusion.  A  RCT  was  included  if  it  enrolled  patients 
with  confirmed  diagnosis  of  lung  cancer,  compared  treatment  versus  no  treatment  (e.g.  supportive 
care,  best  supportive  care,  palliative  care,  placebo  etc.),  and  assessed  overall  survival  as  an  outcome. 

A  study  in  which  patients  had  anticancer  treatment  prior  to  enrollment  and  subgroup  analyses 
were  excluded.  Additionally,  RCTs  comparing  two  active  treatments  were  excluded.  Two  reviewers 
read  the  titles  and  abstracts  of  identified  citations  to  identify  potentially  eligible  studies.  Full  text  of 
potentially  relevant  reports  were  retrieved  and  examined  for  eligibility.  Disagreements  about  study 
inclusion  or  exclusion  were  resolved  via  discussion  until  a  consensus  was  reached. 

Data  Extraction 
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Data  extraction  was  performed  using  a  standardized  data  extraction  form.  Two  reviewers 
independently  extracted  the  following  information  from  each  included  study:  number  of  patients 
enrolled,  number  of  deaths,  median  survival,  funding  source  (industry  versus  public  etc.),  type  of 
centers  involved  (single  versus  multicenter  etc.),  patient  demographics,  patients  baseline  clinical 
characteristics,  and  type  of  control  arm  (for  RCTs  only).  For  cohort  studies,  we  extracted  data  on  the 
number  of  deaths  and  total  number  of  patients  diagnosed  with  lung  cancer.  For  RCTs,  we  extracted 
data  on  the  number  of  deaths  (all-cause  mortality)  and  number  of  participants  randomized  to  the 
control  arm. 

Assessment  of  Methodological  Quality 

To  evaluate  the  methodological  quality  of  included  studies,  a  modified  checklist  of  predefined 
criteria  was  developed  on  four  methodological  domains  pertinent  to  minimization  of  bias.  This 
modified  checklist  uses  applicable  elements  from  existing  tools  (Quality  in  Prognosis  Studies  tool, 
Evidence-Based  Medicine  Group  criteria  for  prognostic  studies,16  Newcastle-Ottawa  Quality 
Assessment  Scale/1  and  Cochrane  Collaboration  risk  of  bias  criteria17)  and  related  studies  (Hudak  et 
al18  and  Altman19).  The  four  domains  included  participation  bias  (extent  to  which  study  sample 
represents  the  population  of  interest  on  key  characteristics),  attrition  bias  (extent  to  which  loss  to 
followup  of  the  sample  was  not  associated  with  key  characteristics),  outcome  measurement  (extent  to 
which  outcome  of  interest  is  adequately  measured  in  study  participants),  data  analysis  and  reporting 
(extent  to  which  statistical  analysis  and  data  reporting  are  appropriate  for  the  study  design).  The 
modified  checklist  contains  1 1  items  for  cohort  studies  and  14  items  for  RCTs.  For  each  item,  a  study 
either  fulfilled  a  certain  criterion  (scored  “Yes”)  or  failed  to  fulfill  the  criterion  (scored  “No”).  To 
assess  methodological  quality  of  studies  included,  we  focused  on  proportion  of  studies  that  fulfilled 
each  quality  criterion  (Table  2). 

Statistical  Analysis 
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Data  synthesis  was  conducted  according  to  the  study  design  separately  as  well  as  combined  in 
the  final  stage  (i.e.,  retrospective  cohort  and  RCT). 

For  the  purpose  of  meta- analysis,  we  used  methods  by  Stuarts  et  al  to  transform  the 
proportions  into  a  quantity  according  to  the  Freeman-Tukey  variant  of  the  arcsine  square  root 
transformed  proportion.  The  pooled  proportion  was  calculated  as  a  back-transform  of  the  weighted 
mean  of  the  transformed  proportions,  using  the  random-effects  model. 

Heterogeneity  of  treatment  effects  between  trials  was  assessed  using  the  I  statistic  with  the 
following  thresholds  for  I2  statistic  values:  low  (25%  to  49%),  moderate  (50%  to  74%),  and  high  (> 
75%).21  We  explored  the  potential  causes  of  heterogeneity  by  assessing  the  differences  between 
subgroups  using  the  test  of  interaction.  We  assessed  robustness  of  the  results  by  conducting  sensitivity 
analysis  with  respect  to  methodological  quality  criteria  of  reporting,  study  location,  and  funding 
source.  RevMan  Version  5.122  was  used  to  perform  the  analyses. 

RESULTS 
Literature  Search 

A  flow  diagram  of  the  literature  search  is  shown  in  Figure  1.  Initial  search  identified  1,562 
potentially  relevant  citations  excluding  71  duplicates.  After  initial  screening  of  titles  and  abstracts, 
1,489  records  were  not  relevant  for  reasons  depicted  in  Figure  1  and  were  excluded.  Further 
assessment  of  full  texts  of  remaining  73  studies  led  to  exclusion  of  51  studies.  Altogether,  22  studies 
met  the  pre-defined  inclusion  criteria:  7  were  retrospective  cohort  studies23'29  and  15  were  RCTs.30'44 
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Figure  1  A  flow  diagram  depicting  the  literature  search  process 
Study  Characteristics 

We  did  not  find  any  inception  cohort  study  or  a  prospective  cohort  study  assessing  prognosis 
of  patients  with  lung  cancer  without  treatment.  The  seven  retrospective  cohort  studies  included  4,418 
patients  and  the  15  RCTs  enrolled  1,031  patients.  Altogether,  the  22  studies  included  5,449  patients. 
All  studies  assessed  prognosis  in  patients  with  NSCLC  and  were  published  between  1973  and  2009 
(Table  1). 

Cohort  Studies:  The  median  sample  size  in  the  cohort  studies  was  131  patients  (range:  39  to 
2,344  patients)  with  a  median  study  period  of  8  years  (range:  5  to  13  years).  Fifty-seven  percent  (4/7) 
and  29%  (2/7)  of  the  studies  reported  number  of  patients  with  stage  I  and  stage  II  NSCLC, 
respectively.  Forty-three  percent  (3/7)  of  the  studies  reported  patients’  cancer  histology.  Seventy-one 
percent  (6/7)  of  the  studies  reported  patient’s  gender.  Forty-three  percent  (3/7)  of  the  studies  reported 
median  age.  Forty-three  percent  (3/7)  of  the  studies  were  conducted  at  single  institutions,  43%  (3/7) 
were  at  multicenter  national  studies,  and  14%  (1/7)  of  the  studies  had  unspecified  study  location. 
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Twenty-nine  percent  (2/7)  of  the  studies  were  publicly  funded,  14%  (1/7)  were  funded  by  both  public 
and  industry,  and  57%  (4/7)  had  not  specified  funding  sources. 

RCTs:  The  median  number  of  patients  enrolled  in  the  RCTs  was  61  patients  (range:  17  to  176 
patients)  with  a  median  study  period  of  3  years  (range:  1  to  7  years).  Median  follow-up  was  reported 
in  33%  (5/15  of  RCTs)  and  ranged  between  2.7  and  43  months.  Seventy-three  percent  (1 1/15)  of  the 
studies  reported  number  of  patients  with  stage  III/IV  NSCLC.  Seventy-three  percent  (13/15)  of  the 
studies  reported  patients’  cancer  histology.  Eighty-seven  percent  (13/15)  of  the  RCTs  reported 
patient’s  gender  and  median  age.  Twenty  percent  (3/15)  of  the  RCTs  were  conducted  at  single 
institutions,  27%  (4/15)  were  at  multicenter  national  studies,  20%  (3/15)  were  at  multicenter 
international,  and  33%  (5/15)  had  unspecified  study  location.  Seven  percent  (1/15)  of  the  RCTs  were 
funded  by  public,  33%  (5/15)  were  funded  by  industry,  7%  (1/15)  were  funded  both  public  and 
industry,  and  53%  (8/15)  had  unspecified  funding  sources. 

Types  of  control  in  RCTs:  Three  studies  described  best  supportive  care  as  comprising 
“symptomatic  or  palliative  treatment  excluding  chemotherapy,”45  “palliative  radiotherapy,  antibiotics, 
and  corticosteroids,”31  “palliative  radiotherapy,  opioid  analgesics,  and  psychosocial  support,”38  or 
“radiation  therapy,  pain  medication,  nutritional  and  psychological  support,  thoracocentesis  and/or  tube 
thorascopy.”44  Three  studies  described  supportive  care  as  comprising  “analgesics,  an  antitussive, 
relief  of  increased  intracranial  pressure,  palliative  radiotherapy,  treatment  of  infections  and  pleural 
effusions,”31  “symptomatic  irradiation  to  involved  fields,”32  or  “palliative  radiation,  analgesics,  and 
psychosocial/nutritional  support.”36  Palliative  care  consisted  of  “radiotherapy,  antibiotics,  coughs 
suppressants,  and  analgesics”34  Symptomatic  treatment  included  “glucocorticosteroids  and  anabolic 
steroids.” 39  No  descriptions  were  provided  for  placebo  and  “no  treatment .” 

Table  1  Characteristics  of  studies  included  in  the  review 


Study  N 

Study 

Disease  Stage 

Histology 

Male 

Median 

period 

(years) 

I  II 

squamous  adeno 

large-cell 

Age 

(years) 

(a)  Cohort  studies 

Raz  2007  1432 

13 

1432  NR 

460  419 

89 

747 

74 
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Wisnivesky  2007| 

2344 

8 

NR 

NR 

NR 

NR 

NR 

1292 

NR 

Chadha  2005 

39 

11 

23 

13 

18 

88 

5 

4 

77 

Henschke  2003 

131 

7 

131 

NR 

NR 

NR 

NR 

NR 

NR 

McGarry  2002| 

49 

5 

NR 

NR 

NR 

NR 

NR 

49 

NR 

Vrdoljak  1994 

130 

7 

55 

56 

61 

35 

34 

120 

60 

Hyde  1973 

293 

8 

NR 

NR 

NR 

NR 

NR 

NR 

NR 

Total/[Range] 

4418 

[5-13] 

1641 

68 

539 

542 

128 

2211 

(b)  RCTs 

III 

IV 

Goss  2009m 

101 

2  [0.23] 

17 

84 

25 

46 

11 

61 

76 

Anderson  2000 

150 

2 

92 

58 

NR 

NR 

NR 

91 

64 

ELVIS  1999  m 

78 

1  [1.08] 

22 

56 

33 

29 

3 

69 

74* 

Cullen  1999  m 

176 

8  [2.17] 

88 

88 

103 

42 

6 

122 

64 

Thongprasert  1999 

98 

4 

49 

49 

31 

49 

12 

NR 

60 

Helsing  1998  m 

26 

5  [3.33] 

3 

23 

5 

17 

4 

18 

65 

Cartei  1993 

50 

7 

NR 

50 

25 

17 

8 

36 

57 

Leung  1992  m 

66 

4  [3.58] 

58 

NR 

31 

18 

7 

48 

62 

Cellerino  1991 

61 

3 

61 

NR 

38 

18 

5 

59 

62 

Quoix  1991 

22 

3 

NR 

22 

NR 

NR 

NR 

NR 

NR 

Kaasa  1991 

43 

3 

NR 

43 

16 

16 

11 

31 

62* 

Ganz  1989 

26 

2 

NR 

26 

9 

17 

NR 

23 

NR 

Rapp  1988 

50 

3 

50 

NR 

12 

24 

12 

38 

58 

Cormier  1982 

17 

2 

17 

NR 

8 

2 

6 

16 

60 

Laing  1975 

67 

2 

15 

20 

23 

5 

9 

59 

64 

Total/[Range] 

1031 

[1-8] 

472 

519 

359 

300 

94 

671 

[57-76] 

Note:  N  =  Sample  size  or  number  of  participants  enrolled;  NR=  data  not  reported;  t  =  Sample  includes  stage  I  and  II 
cancer;  adeno  =  adenocarcinoma;  squamous  =  squamous  cell  carcinoma;  large-cell  =  large-cell  carcinoma; 

*=we  recorded  mean  age  where  median  age  was  not  reported  or  not  extractable,  m  =  median  follow-up  in  parenthesis 


Methodological  Quality 

Cohort:  All  seven  cohort  studies  fulfilled  64%  (7/1 1)  of  the  quality  criteria  (Table  2).  That  is, 
adequate  description  of  population  of  interest  for  key  characteristics,  adequate  description  of  study 
setting/geographic  location,  adequate  participation  in  the  study  by  all  eligible  patients,  reporting  of 
patients  with  missing  data,  a  priori  and  objective  definition  of  outcomes,  and  presentation  of 
frequencies  of  most  important  data  (e.g.,  outcome)  were  reported  in  all  studies.  However,  baseline 
sample  was  adequately  described  for  key  characteristics  in  57%  (4/7)  of  the  studies,  inclusion  and 
exclusion  criteria  were  adequately  described  in  71%  (5/7)  of  the  studies,  follow-up  was  sufficiently 
long  for  outcome  to  occur  in  86%  (6/7)  of  the  studies,  and  alpha  error  and/or  beta  error  were  specified 
a  priori  in  29%  (2/7)  of  the  studies. 

RCTs:  All  15  RCTs  fulfilled  36%  (5/14)  of  the  quality  criteria  (Table  2).  That  is,  adequate 
description  of  population  of  interest  for  key  characteristics,  adequate  description  of  withdrawal 
(incomplete  outcome  data),  a  priori  and  objective  definition  of  outcomes,  and  frequencies  of  most 
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important  data  were  reported  in  all  RCTs.  However,  study  setting  and  geographic  location  were 
adequately  described  in  47%  (7/15)  of  the  RCTs,  baseline  sample  was  adequately  described  for  key 
characteristics  in  93%  (14/15)  of  the  RCTs,  inclusion  and  exclusion  criteria  were  adequately  described 
in  93%  (14/15)  of  the  RCTs,  patients  were  balanced  in  all  aspects  except  the  intervention  in  93% 
(14/15)  of  the  RCTs,  follow-up  was  sufficiently  long  for  outcome  to  occur  in  53%  (8/15)  of  the  RCTs, 
proportion  of  sample  completing  the  study  was  adequate  in  60%  (9/15)  of  the  RCTs,  characteristics  of 
dropouts  versus  completers  was  provided  in  13%  (2/15)  of  the  RCTs,  alpha  error  and/or  beta  error  was 
specified  a  priori  in  47%  (7/15)  of  the  RCTs,  and  data  analysis  was  based  on  intention  to  treat  analysis 
principle  in  53%  (9/15)  of  the  RCTs. 


Table  2  Methodological  Quality  of  Lung  Cancer  prognosis  Studies 


Study  Design/Domain/Criterion 

Criteria  fulfilled 

n/N 

% 

Cohort  studies  (11  items) 

Participation  bias 

A  Population  of  interest  is  adequately  described  for  key  characteristics15 

7/7 

100 

B  Study  setting  and  geographic  location  is  adequately  described 

7/7 

100 

C  Baseline  sample  is  adequately  described  for  key  characteristics15 

4/7 

57 

D  Inclusion  and  exclusion  criteria  are  adequately  described 

5/7 

71 

E  There  is  adequate  participation  in  the  study  by  all  eligible  patients15 

7/7 

100 

Attrition  bias 

F  Follow-up  is  sufficiently  long  for  outcome  to  occur  (>  6  months)16 18 19  46 

6/7 

86 

G  Patients  with  missing  data  were  reported 

7/7 

100 

Outcome  measurement 

H  Definition  of  outcome  is  provided  a  priori  15 

7/7 

100 

I  Obj  ective  definition  of  outcome  is  provided1 

7/7 

100 

Data  analysis  and  reporting 

J  Alpha  error  and/or  beta  error  is  specified  a  priori 

2/7 

29 

K  Frequencies  of  most  important  data  (e.g.,  outcomes)  are  presented18 19  47 

7/7 

100 

Randomized  Controlled  Trials  (15  items) 
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Participation  bias 

L  Population  of  interest  is  adequately  described  for  key  characteristics15 

15/15 

100 

M  Study  setting  and  geographic  location  is  adequately  described 

7/15 

47 

N  Baseline  sample  is  adequately  described  for  key  characteristics15 

14/15 

93 

O  Inclusion  and  exclusion  criteria  are  adequately  described 

14/15 

93 

P  Patients  were  balanced  in  all  aspects  except  the  intervention 

15/15 

93 

Attrition  bias 

Q  Follow-up  is  sufficiently  long  for  outcome  to  occur  (>  6  months)16  18  19  46  48 

8/15 

53 

R  Proportion  of  sample  completing  the  study  is  adequate  (>80%)15,16, 18,47,49,50 

9/15 

60 

S  Description  of  withdrawal  (incomplete  outcome  data)  is  provided 

15/15 

100 

T  Characteristics  of  dropouts  versus  completers  is  provided 

2/15 

13 

Outcome  measurement 

U  Definition  of  outcome  is  provided  a  priori 15 

15/15 

100 

V  Obj  ective  definition  of  outcome  is  provided1 

15/15 

100 

Data  analysis  and  reporting 

W  Alpha  error  and/or  beta  error  is  specified  a  priori 

7/15 

47 

X  Data  analysis  was  based  on  intention  to  treat  analysis  principle 

9/15 

53 

Y  Frequencies  of  most  important  data  (e.g.,  outcomes)  are  presented18,19,47 

15/15 

100 

Mortality 


Cohort:  Data  on  mortality  was  extractable  from  all  seven  cohort  studies  enrolling 


4,418  patients.  As  shown  in  Figure  2,  the  pooled  proportion  of  mortality  for  patients  without 


anticancer  treatment  was  0.97  (95%CI:  0.96  to  0.99).  There  was  a  statistically  significant 


heterogeneity  among  pooled  cohort  studies  (I2  =93%,  P  <  0.00001). 


RCTs:  Data  on  mortality  was  extractable  from  the  control  arm  of  all  15  RCTs  (1 ,031 


patients).  Figure  2  shows  that  the  pooled  proportion  of  mortality  for  patients  in  the  control 


arm  (without  active  treatment)  was  0.96  (95%  Cl:  0.94  to  0.98).  There  was  a  statistically 


significant  heterogeneity  among  pooled  control  arm  of  RCTs  (I2  =80%,  P  <  0.00001). 
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Combined  (Cohort  and  RCTs):  Pooled  proportion  of  mortality  across  the  22  studies 


was  0.97  (95%CI:  0.96  to  0.98).  Because  these  two  designs  are  inherently  different  from 
each  other,  we  conducted  separate  analyses.  However,  as  shown  in  Figure  2,  test  for 
subgroup  differences  showed  no  statistically  significant  heterogeneity  between  the  two  study 
designs  (P  =  0.28). 
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Total  participants  Weight  Proportion,  95%CI 


Study  or  Subgroup 

n 

Chadha  2005 
fienschke  2003 
Hyde  1973_ 

Mte@©irf^e2002 

efer 

Vrdoljak  1994 
Wisnivesky  2007 

Subtotal  (95%  Cl) 

Heterogeneity:  Tau2  =  0.00;  Chi2  =  84.47,  df  =  6  (P  <  0.00001);  I 
Test  for  overall  effect:  Z  =  3.57  (P  =  0.0004) 

RCTs 


93% 


Anderson  2000 

150 

3.7% 

0.93 

[0.89, 

0.98] 

Cartei  1993 

50 

9.0% 

1.00 

[0.98, 

1.02] 

Cellerino  1991 

61 

0.5% 

0.77 

[0.67, 

0.89] 

Cormier  1982 

17 

3.0% 

0.99 

[0.94, 

1.04] 

Cullen  1999 

176 

10.8% 

0.99 

[0.98, 

1.00] 

ELVIS  1999 

78 

7.4% 

0.99 

[0.96, 

1.01] 

Ganz  1989 

26 

1.7% 

0.96 

[0.90, 

1.03] 

Goss  2009 

101 

11.1% 

1.00 

[0.99, 

1.01] 

Helsing  1998 

26 

0.7% 

0.92 

[0.82, 

1.04] 

Kaasa  1991 

43 

1 .2% 

0.93 

[0.85, 

1.02] 

Laing  1975 

67 

0.9% 

0.85 

[0.77, 

0.94] 

Leung  1992 

66 

0.4% 

0.68 

[0.58, 

0.81] 

Quoix  1991 

22 

1 .3% 

0.95 

[0.88, 

1.04] 

Rapp  1988 

50 

2.2% 

0.96 

[0.90, 

1.02] 

Thongprasert  1999 

98 

1 .5% 

0.87 

[0.80, 

0.94] 

Subtotal  (95%  Cl) 

1031 

55.3% 

0.96 

[0.94, 

0.98] 

Heterogeneity:  Tau2  = 

0.00;  Chi2  = 

69.51,  df  = 

14  (P<  0.00001);  l2  = 

=  80% 

Test  for  overall  effect: 

Z  =  4.00  (P 

<0.0001) 

Total  (95%  Cl) 

5449 

100.0% 

0.97 

[0.96, 

0.98] 

Heterogeneity:  Tau2  = 

0.00;  Chi2  = 

160.37,  df 

=  21  (P<0 

.00001);  I2 

=  87°/ 

D 

Test  for  overall  effect: 

Z  =  5.16  (P 

<0.00001) 

Test  for  subgroup  differences:  Chi: 

2=  1.14,  df 

=  1  (P  =  0.28),  l2=  12 

.6% 

Proportion 
IV,  Random,  95%  Cl 


39 

2.1% 

0.87  [0.82,  0.93] 

■ 

131 

2.1% 

0.87  [0.82,  0.93] 

293 

5.5% 

0.96  [0.93,  0.99] 

■ 

49 

0.5% 

0.82  [0.71,  0.94] 

■ 

1432 

10.7% 

0.97  [0.96,  0.98] 

■ 

130 

11.7% 

1.00  [0.99,  1.01] 

2344 

12.1% 

1.00  [1.00,  1.00] 

i 

4418 

44.7% 

0.97  [0.96,  0.99] 

0.5  0.7  1  1.5  2 


Figure  2  Pooled  proportion  of  mortality  in  lung  cancer  studies.  The  size  of  each  square  is 


proportional  to  the  weight  of  the  study  (inverse  variance) 


Sensitivity  Analysis 


To  assess  the  robustness  of  overall  results  according  to  the  study  design  (cohort  vs.  RCT)  as  well  as 


explore  the  reasons  for  observed  heterogeneity  in  the  pooled  proportion  of  mortality,  we  conducted 


additional  sensitivity  analyses.  For  both  cohort  studies  and  RCTs,  we  conducted  sensitivity  analyses 
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according  to  methodological  quality  criteria,  funding  source,  and  study  location.  For  RCTs  only,  we 
conducted  additional  sensitivity  analyses  according  to  type  of  control.  The  results  of  sensitivity 
analyses  are  summarized  in  Figure  3.  Overall,  the  results  remained  unchanged  in  the  sensitivity 
analyses.  There  were  no  statistically  significant  differences  in  the  proportion  of  mortality. 

Cohort:  In  cohort  studies,  there  was  no  statistically  significant  difference  in  the  proportion  of 
mortality  according  to  any  methodological  criteria  of  reporting.  With  respect  to  study  location,  the 
pooled  proportion  of  mortality  in  cohort  studies  conducted  at  multicenter  national  locations  was  0.95 
(95%CI:  0.89  to  1.01)  and  at  single  institution  was  0.98  (95%CI:  0.95  to  1.01)  whereas  the  pooled 
proportion  of  mortality  in  cohort  studies  conducted  at  unspecified  locations  was  0.87  (95%CI:  0.82  to 
0.93).  Test  for  overall  interaction  among  these  subgroups  was  statistically  significant  (P  =  0.007). 
Regarding  funding  source,  the  pooled  proportion  of  mortality  in  public-funded,  unspecified  funding 
sources,  and  public/industry- funded  cohort  studies  were  1.00  (95%CI:  1.00  to  1.00),  1.00  (95%CI: 
0.99  to  1.00),  and  0.97  (95%CI:  0.96  to  0.98),  respectively.  The  test  for  overall  interaction  among 
these  subgroups  was  statistically  significant  (P  <  0.0001). 

RCTs:  There  was  no  statistically  significant  difference  in  the  proportion  of  mortality 
according  to  methodological  criteria  of  reporting,  study  location,  and  funding  source.  With  respect  to 
type  of  control,  the  pooled  proportion  of  mortality  in  RCTs  involving  best  supportive  care,  no 
treatment,  placebo,  supportive  care,  and  symptomatic  treatment  as  control  were  0.90  (95%CI:  0.83  to 
0.97)  and  in  RCTs  involving  supportive  care  as  control  was  0.96  (95%CI:  0.92  to  1.00),  0.86  (95%CI: 
0.81  to  0.92),  1.00  (95%CI:  0.99  to  1.01),  0.96  (95%CI:  0.92  to  1.00),  and  0.97  (95%CI:  0.92  to  1.03), 
respectively.  Test  for  overall  interaction  among  these  subgroups  was  statistically  significant  (P  < 
0.00001). 
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Proportion 

Subgroup _ Number  of  studiesflpartici  pants)  Proportion,  95%  Cl  IV,  Random,  95%  Cl 

Study  location  (Cohort  studies) 


Multicenter  national 

3/  (2768) 

0.95  [0.89,  1.01] 

Single  institution 

3/(1116) 

0.98  [0.95,  1.01] 

Unspecified  location 

Heterogeneity  between  sub-groups:!2  =  80.1% 

1/(39) 

0.87  [0.82,  0.93] 

Funding  source  (Cohort  studies) 

Public 

2/  (2637) 

0.98  [0.95,  1.02] 

Public/Industry 

1/(1432) 

0.97  [0.96,  0.98] 

Unspecified  source 

Heterogeneity  between  sub-groups:!2  =  94% 

4/ (349) 

1.00  [0.99,  1.01] 

Study  location  (RCTs) 

Multicenter  international 

3/ (329) 

0.98  [0.95,  1.01] 

Multicenter  national 

4/  (313) 

0.94  [0.88,  1.00] 

Single  institution 

3/(163) 

0.91  [0.86,0.97] 

Unspecified  location 

Heterogeneity  between  sub-groups:!2  =  55.8% 

5/  (226) 

0.91  [0.84,0.99] 

Funding  source  (RCTs) 

Industry 

5/(551) 

0.97  [0.95,  1.00] 

Public 

1/  (67) 

0.85  [0.77,  0.94] 

Public/Industry 

1/ (50) 

0.96  [0.90,  1.02] 

Unspecified  source 

Heterogeneity  between  sub -groups: I2  =  57.6% 

8/  (363) 

0.95  [0.91,0.99] 

Type  of  control  (RCTs) 

Best  supportive  care 

5/  (314) 

0.90  [0.83,0.97] 

No  treatment 

2/(165) 

0.86  [0,81,0.91] 

Placebo 

2/(118) 

1.00  [0.99,  1.01] 

Supportive  care 

4/  (215) 

0.96  [0.92,  1.00] 

Symptomatic  treatment 

2/  (219) 

0.97  [0.92,  1.03] 

Heterogeneity  between  sub-groups:!2  =  87% 

I - 1 - - 1 - 1 

0.01  0.1  1  10  100 


Figure  3  Pooled  Proportions  of  Mortality  and  Heterogeneity  Between  Subgroups 


DISCUSSION 
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This  is  the  first  study  to  provide  most  comprehensive  data  related  to  natural  history  of  lung  cancer. 

The  results  show  that  prognosis  of  patients  with  lung  cancer  not  receiving  treatment  is  very  high. 
Regardles  of  the  study  design  (i.e.  cohort  versus  RCTs)  the  findings  were  similar  and  did  not  differ 
according  to  disease  severity.  For  example,  all  cohort  studies  assessed  mortality  in  patients  with  early 
stage  NSCLC  (stage  I/II)  and  all  RCTs  enrolled  patients  with  advance  stage  NSCLC  (stage  III/IV). 
However,  the  mortality  rates  from  cohort  and  RCTs  essentially  remained  unchanged  (97%  vs  96%). 
Overall,  included  studies  were  of  moderate  methodological  quality. 

The  findings  from  our  study  is  similar  to  the  study  by  Detterbeck  and  Gibson  which  showed  a 
98%  5-year  mortality  rate  for  stage  I/II  lung  cancer  (median  survival  =  10  months).  Despite  the 
obvious  similarity  in  results  our  study  is  significantly  different  in  the  conduct  and  analysis.  For 
example,  the  study  by  Detterbeck  and  Gibson  did  not  employ  a  systematic  approach  to  data 
collection  and  analysis  (i.e.  not  a  systematic  review)  and  therefore  the  findings  are  not  reproducible. 
The  similarity  in  findings  might  be  an  artifact  of  play  of  chance.  Furthermore,  quantitative  synthesis  of 
results  across  included  studies  was  not  performed  in  the  study  by  Detterbeck  and  Gibson4  which  was 
undertaken  in  our  study.  Another  unique  feature  of  our  study  lies  in  the  inclusion  of  RCTs  in  addition 
to  retrospective  studies.  None  of  the  previous  studies  on  the  topic  have  utilized  the  approach  of 
pooling  data  from  one  arm  of  RCTs  for  accurate  assessment  of  prognosis.  Therefore,  due  to  the 
reasons  enumerated  here  the  study  presented  here  is  the  most  comprehensive  to  date  reporting  the 
natural  history  of  lung  cancer. 

Our  study  has  some  limitations.  For  example,  we  observed  a  statistically  significant 
heterogeneity  in  pooled  results  which  we  could  not  explain  through  subgroup  analyses.  We  suspect 
that  the  observed  heterogeneity  is  clinical  and  not  methodological.  Specifically  in  the  case  of  RCTs, 
the  constitution  of  control  arm  varied  across  pooled  studies.  For  example,  five  RCTs  employed  best 
supportive  care  as  control,  four  had  supportive  care,  two  had  placebo,  two  had  no  treatment  and 
another  two  had  symptomatic  treatment  as  control.  While,  the  definitions  are  very  clear  on  placebo 
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and  no  treatment,  which  was  also  explained  by  the  sensitivity  analyses  (I2  =0%  for  both  subgroups), 
the  composition  of  best  supportive  care,  supportive  care,  and  symptomatic  treatment  varied 
significantly  across  pooled  studies.  In  these  cases,  the  observed  heterogeneity  remained  unexplained. 
The  findings  are  also  limited  in  terms  of  generalizability  by  the  fact  that  all  included  studies  enrolled 
patients  with  NSCLC  due  to  which  the  results  are  not  entirely  applicable  to  all  lung  cancers.  However, 
it  is  important  to  note  that  a  systematic  review  is  limited  by  the  availability  of  data  and  we  did  include 
all  available  data  related  to  prognosis  of  lung  cancer  patients  without  treatment. 

Comprehensive  data  on  the  natural  history  of  lung  cancer  is  required  for  informed  decision 
making  by  patients,  physicians  and  researchers.  For  patients,  it  serves  as  the  basis  for  their  expected 
outcome  with  and  without  treatment,  which  is  critical  in  cases  of  diseases  with  high  mortality.  For 
physicians,  accurate  and  reliable  information  facilitates  shared  decision  making  with  patients  related 
to  choice  of  interventions  or  no  intervention.  Most  importantly,  the  findings  are  needed  by  researchers 
to  avoid  optimism  bias.51  Briefly,  optimism  bias  refers  to  unwarranted  belief  in  the  efficacy  of  new 
therapies.  A  study  by  Djulbegovic  et  al. 51  assessed  the  role  of  optimism  bias  in  a  cohort  of  trials 
conducted  by  the  National  Cancer  Institute  Cooperative  Groups  and  concluded  that  the  optimism  bias 
is  the  primary  reason  for  inconclusive  findings  in  the  context  of  RCTs.  Accordingly,  the  results  from 
our  study  will  help  researchers  determine  the  most  optimal  rate  of  expected  improvement  in  mortality 
with  innovative/newer  treatments. 
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Prognostat  is  an  interactive  Web-based  prognostic  tool  for  estimating  hospice  patient  survival  based 
on  a  patient’s  palliative  performance  scale  score,  age,  gender,  and  cancer  status.  The  tool  was 
developed  using  data  from  5,893  palliative  care  patients  collected  at  the  Victoria  Hospice  since  1994. 
This  study  externally  validates  Prognostat  on  a  retrospective  cohort  of  590  hospice  patients  in 
Lifepath  Hospice  Center,  Florida,  USA.  The  criteria  used  to  evaluate  the  prognostic  performance  are 
the  Brier  score,  area  under  the  receiver  operating  curve,  discrimination  slope  and  Hosmer-Lemeshow 
goodness-of-fit  test.  Though  the  Kaplan-Meier  curves  show  each  PPS  level  to  be  distinct  and 
significantly  different,  the  findings  reveal  poor  performance  of  the  prognostic  tool  in  our  cohort  of 
hospice  patients.  Before  redeveloping  a  new  prognostic  model  researchers  are  encouraged  to  combine 
survival  estimates  obtained  using  Prognostat  with  the  information  from  their  cohort  of  patients.  To 
that  end,  Prognostat  needs  to  explicitly  report  patient  risk  scores  to  be  useful  to  clinicians. 

Keywords:  Prognostat,  Survival  prognostication,  palliative  performance  scale,  external  validation 
VI.  INTRODUCTION 

Accurate  prognostication  of  survival  of  patients  in  hospice  care  gives  patients  and  their  family 
members  a  vital  opportunity  to  attend  to  matters  such  as  planning,  prioritizing,  and  preparing  for 
death[l].  Predicting  patient  survival  is  a  complex  decision  making  process  which  is  often  affected  by 
optimism  or  avoidance  that  may  lead  to  poor  prediction  of  life  expectancy  and  overestimating  survival 
at  times  by  a  factor  of  five[2].  Successful  prognostication  of  patient  survival  depends  on  developing 
and  testing  prognostic  models,  which  entails  having  accurate  patient  data  for  prognosis  and  selecting 
clinically  relevant  candidate  predictors  and  measure(s)  of  model  performance,  usually  in  the  context  of 
a  multivariable  regression  survival  model[3].  This  process  produces  patient  performance  scores  that 
allow  for  classification  of  patients  into  different  risk  groups. 

The  usefulness  and  validity  of  a  prognostic  model  is  judged  by  how  well  it  performs  for 
patients  that  come  from  different  centers[4].  Validating  a  prognostic  model  is  generally  accepted  to 
mean  that  given  a  patient  population  it  works  in  a  data  set  other  than  the  one  which  is  used  to  develop 
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it  [4,  5],  It  is  also  generally  accepted  that  the  validation  process  should  follow  guidelines  and  that  un¬ 
validated  prognostic  models  should  not  be  applied  in  clinical  practice  [6-8].  As  the  value  of  any 
prediction  model  is  its  generalizabilty  to  other  groups  of  patients,  our  goal  is  to  externally  validate 
Prognostat  [9],  a  Web-based  interactive  prognostic  tool  for  estimating  hospice  patient  survival,  on  a 
retrospective  cohort  of  590  hospice  patients  in  Florida,  USA.  Prognostat  gives  estimates  of  survival 
based  on  the  palliative  patient’s  age,  gender,  diagnosis  and  palliative  performance  scale  (PPS)  score 
[10],  which  has  been  established  as  an  accurate  measure  of  patient  survival  in  the  palliative  setting 
[11-15], 

In  the  next  section  we  discuss  Prognostat  and  introduce  the  measures  of  model  performance.  Since 
predictive  performance  may  decrease  when  Prognostat  is  tested  in  new  patients  compared  to  the 
performance  estimated  in  the  patients  used  to  develop  the  model,  we  also  discuss  strategies  for 
updating  Prognostat. 

VII.  METHODS 

Study  Sample  and  Survival  Estimation  using  Prognostat 

The  patient  data  were  obtained  from  the  Lifepath  Hospice  and  Palliative  Care  center,  licensed  since 
1983  to  serve  in  Hillsborough  County,  Florida.  The  data  regarding  590  consecutive  deceased  patients 
was  extracted  starting  January  2009  and  going  backwards.  This  study  was  a  retrospective  review  of 
deceased  patients’  medical  records  and  only  data  that  pertained  to  outcomes  was  collected;  personal 
information  was  not  collected  and  the  data  were  de-identified  prior  to  analysis.  The  study  was 
approved  by  the  University  of  South  Florida  Institutional  Review  Board.  Two  research  assistants 
extracted  all  data  necessary  to  populate  the  model  variables  and  two  faculty  members  randomly 
checked  25%  of  the  data  for  accuracy. 

Prognostat  is  a  web-based  interactive  prognostic  tool  developed  at  the  University  of  Victoria 
using  survival  estimates  of  5,893  palliative  care  patients  collected  at  the  Victoria  Hospice  starting  in 
1994.  The  variables  or  covariates  found  to  be  significant  predictors  of  patient  survival  were  the 
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patient’s  gender  (male  vs  female),  age  group  (19-44,  45-64,  65-74,  75-84,  85+),  diagnosis  (Lung 
Cancer,  Breast  Cancer,  Colorectal  Cancer,  Prostate  Cancer,  Other  Cancer,  All  Cancer,  Non-cancer 
Illness)  and  PPS. 

Palliative  performonance  scale  was  developed  and  reported  by  Anderson  et  al.  [10]  as  a 
measure  of  functional  status  of  patients  receiving  palliative  care.  The  scale  has  1 1  possible  mutually 
exclusive  levels,  from  PPS  of  0%  (the  patient  is  dead)  to  PPS  of  100%  (the  patient  is  ambulatory  and 
healthy).  Numerous  studies  have  studied  its  prognostic  accuracy  of  survival  in  a  variety  of  settings  and 
found  it  provides  meaningful  estimates  of  patient  survival  [11,  12,  14-20].  PPS  has  been  found  to  be 
both  valid  and  reliable[21], 

Prognostat  survival  estimates  were  derived  using  the  Cox  proportional  hazards  (CPH)  model. 
For  a  vector  of  covariates  x  and  parameter  vector  p,  the  survival  function  S  (t;  x)  for  the  CPH  model  is 
commonly  expressed  as 

S(t;  x)  =  [S0  (t)]exP^ 

where  So  (t)  is  the  baseline  survival  function,  i.e.  survival  function  when  the  all  the  covariates  x  are 
equal  to  zero.  In  the  CPH  framework,  the  estimation  of  the  (linear)  prognostic  index  xp  does  not 
require  the  formulation  of  the  baseline  cumulative  survival  function  So  (t),  which  itself  can  be 
estimated  conditional  on  the  covariate  estimates  using  the  Breslow  and  Kalbfleisch-Prentice 
estimators [22],  However,  the  full  parametric  estimation  of  So  (t)  is  not  possible,  which  makes 
prediction  of  baseline  survival  from  the  primary  to  the  secondary  data  set  not  viable.  At  present, 
Prognostat  does  not  explicitly  report  prognostic  indices  xp,  which  makes  model  calibration  in  other 
populations  unfeasible. 

Assessment  of  Model  Performance 

We  analyzed  Prognostat ’s  predictive  performance  based  on  the  ability  of  the  estimated  risk  score  to 
predict  survival  based  on  measures  of  accuracy,  calibration  and  discrimination.  Accuracy  refers  to  the 
difference  between  the  predicted  probability  of  survival  and  observed  patient  survival.  The  Brier 
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score  is  a  quadratic  scoring  rule  that  calculates  the  differences  between  the  actual  outcomes  and 
predicted  probabilities[23].  Given  the  predicted  probability  of  survival  p;  at  time  t  for  patient  i,  and  Y, 
binary  (0-1,  dead-alive)  variable,  the  Brier  score  is  defined  as  £j(Yj  (1  —  pj)2  +  (1  —  Yj)  p,2).  A 
Brier  score  of  0  indicates  a  perfect  model,  while  0.25  indicates  a  non-informative  model  (the  value 
achieved  when  issuing  a  predicted  probability  of  50%  to  each  patient).  The  Brier  score  may  be  scaled 

(Brier  \ 

i  —  — - )  100% 

Briermax/ 

which  has  interpretation  similar  to  the  Pearson  correlation  coefficient[24]. 

Calibration  refers  to  how  closely  the  predicted  survival  at  a  pre-specified  time  agrees  with  the 
observed  survival.  Since  calibration  is  essentially  a  test  of  fit,  we  also  applied  the  Hosmer-Lemeshow 
(HL)  test[25]  on  the  dead  versus  alive  binary  outcome.  The  HL  Chi-square  statistic  involves  grouping 
of  the  observations  (most  commonly  in  deciles)  based  on  the  predicted  probabilities  and  then  testing 
the  hypothesis  that  the  difference  between  observed  and  predicted  events  is  simultaneously  zero  for  all 
the  groups.  This  test  is  equivalent  to  testing  the  hypothesis  that  the  observed  number  of  events  in  each 
of  the  groups  is  equal  to  the  expected  number  of  events  based  on  the  fitted  model.  The  higher  the  HL 
p-value,  the  better  calibrated  the  model  is.  The  HL  calibration  can  be  visually  expressed  by  plotting 
deciles  of  predicted  versus  observed  proportions  of  survival  at  each  time  point. 

Discrimination  is  the  ability  of  the  model  to  differentiate  between  the  patients  who  died  versus 
those  who  survived  at  a  pre-specified  time.  A  rank  order  statistic  commonly  used  to  summarize 
discrimination  with  and  without  the  outcome  has  been  the  area  under  the  receiver  operating  curve 
(AUC)[26],  which  is  a  plot  of  the  sensitivity  (true  positive  rate)  against  1  -  specificity  (false  positive 
rate)  for  consecutive  cutoffs  of  the  probability  of  an  outcome.  The  maximum  value  of  AUC  =  1 
indicate  a  perfect  prediction  model,  while  a  value  of  AUC  =  0.5  indicates  that  50%  of  the  patients 
have  been  correctly  classified  (as  good  as  by  chance).  As  a  rank  order  statistic,  AUC  is  insensitive  to 
errors  such  as  difference  in  average  survival. 
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The  discrimination  slope  is  a  measure  of  how  well  the  subjects  with  and  without  the  outcome  are 
separated.  It  is  defined  as  the  absolute  difference  in  mean  predictions  of  survival  (mean[p;])  between 
those  who  died  and  those  who  survived  at  time  t[5].  In  addition  to  the  discrimination  slope,  we  have 
assessed  the  extent  to  which  survival  differentiation  at  each  time  point  is  achieved  using  box  plots.  All 
statistical  calculation  were  performed  using  Stata  version  1 1.2[27] 

VIII.  RESULTS 

The  patient  characteristic  of  the  retrospective  cohort  are  summarized  in  Table  1.  In  addition  to  our 
cohort  of  590  patients,  in  each  column  we  present  data  information  given  in  Prognostat  (where 
reported)  as  a  second  cell  entry.  Even  though  the  overall  median  survival  times  were  not  significantly 
different  (6  vs.  8  days),  the  table  shows  significant  discrepancies  in  the  distribution  of  percentages  for 
age  and  cancer  status.  There  is  also  a  significant  discrepancy  in  the  distribution  of  percentages  and 
median  survival  times  for  PPS. 

For  our  cohort,  the  Kaplan-Meier  curves  stratified  by  initial  PPS  level  are  shown  in  Figure  1. 
The  curves  show  good  separation  indicating  that  the  different  risk  groups  are  well  defined.  Fifteen 
patients  with  PPS  score  of  60%  were  dropped  due  to  the  crossing  of  the  Kaplan-Meier  estimate  of  PPS 
50%.  The  log-rank  test  for  equality  of  survival  curves  was  highly  significant  at  P  =  0.001  for  PPS  and 
cancer  status,  but  not  for  age  and  gender.  Likewise,  when  adjacent  categories  of  PPS  were  compared 
(PPS  10%  vs  20%,  20%  vs  30%, ..etc),  pair-wise  log-rank  tests  were  all  significant  at  P  =  0.05  level, 
except  for  PPS  40%  vs  PPS  50%  (P  =0.394),  due  to  initial  crossing  of  two  survival  curves  and  longer 
tail  of  the  PPS  40%  group.  For  our  patients,  the  CPH  model  indicates  that  age  and  cancer  status  were 
not  significantly  related  to  the  hazard  for  death.  Patients  who  were  44  years  old  and  younger  did  not 
have  significantly  lower  hazard  than  other  age  groups,  nor  did  male  patients  compared  with  females 
(Table  1). 

The  measures  of  accuracy,  discrimination  and  calibration  for  days  1,  2,  4,  7,  14  and  30  are 
given  in  Table  2  and  show  poor  performance  of  Prognostat  overall.  The  discrimination  slopes  are  low 
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and  the  HL  p-values  significant  for  all  six  days  of  measurement,  and  indicate  poor  calibration.  In  the 
HL  calibration  plot  of  predicted  versus  observed  proportion  of  those  who  survived  (Figure  2B)  circles 
are  mostly  unaligned  with  the  45  degree  line.  They  show  that  in  our  cohort  of  patients  Prognostat 
consistently  underestimates  survival  for  days  1,  2,  4,  7  and  14,  and  overestimates  for  day  30.  The 
larger  circles  indicate  that  these  points  are  based  on  more  data.  The  absence  of  circles  in  any  given 
decile  indicates  that  there  were  no  predictions  in  that  interval.  The  overlapping  box  plots  (Figure  2A) 
confirm  poor  discrimination. 

IX.  DISCUSSION 

In  this  manuscript  we  externally  validated  a  web-based  interactive  prognostic  tool  Prognostat  and 
found  it  performed  poorly  in  our  cohort  of  palliative  patients.  It  is  not  uncommon  for  the  predictive 
performance  of  a  model  to  be  decreased  when  it  is  tested  in  new  patients  compared  to  its  performance 
in  the  cohort  of  patients  used  to  develop  the  model,  as  patient  populations  will  likely  differ.  This  has 
been  recognized  in  case  of  PPS  possibly  due  to  the  differences  in  patient  cohort  characteristics, 
location  of  care,  and  misinterpretation  regarding  how  to  use  the  performance  tool  and  the  inter¬ 
reviewer  discrepancy?  [15,  28],  The  difference  between  our  cohort  and  the  population  used  to  develop 
Prognostat  is  pronounced  for  age  at  treatment,  cancer  status  and  PPS  score. 

However,  instead  of  re-developing  a  new  model,  knowledge  from  previous  studies  should  be 
used  to  update  the  existing  prediction  model  using  shrinkage  and  recalibration  methods  [29,  30],  The 
updating  methods  can  range  from  adjustments  of  the  baseline  survival  to  adjustment  of  predictor 
weights  using  adjustment  factors.  This  may  entail  re-estimating  predictor  weights  and  adding  new 
predictors  or  removing  existing  predictors  from  the  original  model[7].  Ideally,  the  updated  model 
should  also  be  externally  validated.  For  Prognostat  to  be  useful  to  hospice  and  palliative  care 
researchers,  it  should  report  explicit  risk  scores  to  be  combined  with  new  patient  information,  as  well 
as  guidance  on  how  this  should  be  done. 
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Prognostat  is  also  restricted  in  the  framework  of  the  Cox  proportional  hazards  model, 
especially  due  to  the  fact  that  it  is  impossible  to  directly  model  and  report  the  baseline  survival 
function.  This  is  essential  in  calibrating  survival  estimates  in  a  new  population  of  patients.  We  have 
found  that  Royston-Parmar  family  of  survival  functions[31]  is  more  accurate  and  flexible  than  the 
CPH  model[32]  by  allowing  for  parametric  modeling  of  the  baseline  survival  function  and  relaxing  of 
the  proportional  hazards  assumption. 

A  limitation  of  our  study  is  that  it  was  confined  to  external  validation  of  an  existing  model, 
which  needs  to  be  re-calibrated  and  tested  prospectively  on  an  independent  data  set  from  our  patient 
population.  Without  explicit  information  from  Prognostat  regarding  patient  risk  scores  or  linear 
predictors,  this  is  not  feasible  at  this  time. 
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Figure  and  Table  Captions 

Figure  1.  Kaplan-Meier  survival  curves  by  initial  PPS. 

Figure  2.  Box  plot  showing  predictions  by  actual  outcome  (survival)  for  hospice  patients  (Figure  2A). 
Hosmer-Lemeshow  calibration  plot  of  predicted  versus  actual  proportion  of  survival  (2B).  The  45° 
degree  line  denotes  the  perfect  agreement  between  predicted  and  observed  survival. 

Table  1.  Patient  characteristics  and  survival  time  by  age,  gender,  cancer  diagnosis  and  PPS.  The 
second  cell  entry  (where  available)  marked  with  *  is  from  the  Victoria  Hospice  cohort  of  patients  used 
to  develop  Prognostat. 

Table  2.  Prognostat  performance  measures.  AUC  =  Area  under  the  receiver  operating  curve.  HL  = 
Hosmer-Lemeshow. 

Figure  1 
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1 

pps  20% 

16(8.  24) 

5(4.6).  2(1  8,2  2)* 

1-371 

125(21  7%).  732  (12  5%)* 

0  52*  (0  41*.  0  664),-  0  001 

PPS  30°o 

15(111*'. 

7  (5.9),  5  (4.6. 5.4)* 

1-140 

123(214*®).  1403(23  9*o)* 

0  402(0  318.0  508).-  0001 

PPS40*o 

24(18.30) 

14(11.17).  13(11.9. 14  1)* 

1-174 

96(16.7%).  1590(27  0*o)* 

0.271(0.209.0.351).'  0  001 

PPS  50 

30(22.38) 

18(14.45).  28(25  1.309)* 

1-76 

29(5  0*o).  1003(1*1*.)* 

0  223(0149.0  334).-  0001 

PPS  60% 

23(14.32) 

18(5,  29).  43  (37  9.481)* 

4-61 

15(2  6*o).453(7  7*0)* 

0  2*1(0  159.0  461).  <0001 

Day  7 


Day  14 


Fitted  pfcpatiofi 
Day? 


Figure  2 

Table  1 .  Survival  time  by  age.  itndir.  diagnosis  and  intltlal  PPS 


prwtttiM  propatiM 
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Day  2 


Day  4 
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Table  2.  Accuracy,  discrimination  and  calibration  measures  for  model  performance  in  predicting 
patient  survival  using  Prognostat 

Day  (%  patients  survived) 

Score 

PPS  adjusted 

Day  1  (89.4%) 

Brier 

Brier  Scaled 

AUC  (95%  Cl) 

Slope 

HL  p-value 

0.099 

18.30% 

0.699(0.628,0.771) 

0.119 

0.0023 

Day  2  (77.6%) 

Brier 

Brier  Scaled 

AUC 

Slope 

HL  p-value 

0.187 

18.40% 

0.741  (0.689,0.791) 

0.234 

<0.001 

Day  4  (59.7%) 

Brier 

Brier  Scaled 

AUC  (95%  Cl) 

Slope 

HL  p-value 

0.244 

0.40% 

0.724(0.682,0.767) 

0.243 

<0.001 

Day  7  (42.4%) 

Brier 

Brier  Scaled 

AUC  (95%  Cl) 

Slope 

HL  p-value 

0.211 

2.20% 

0.764(0.724,0.803) 

0.263 

<0.001 

Day  14(24%) 

Brier 

Brier  Scaled 

AUC  (95%  Cl) 

Slope 

HL  p-value 

0.155 

5.50% 

0.756(0.709,0.804) 

0.203 

<0.001 

Day  30  (10.8%) 

Brier 

Brier  Scaled 

AUC  (95%  Cl) 

Slope 

HL  p-value 

0.088 

39.40% 

0.758(0.696,0.820) 

0.136 

<0.001 

Cl:  confidence  interval,  HL:  Hosmer-Lemeshow  statistic,  AUC: 

area  under  the  receiver  operating  curve 
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External  Validation  of  Prognostic  Models  in  Terminally  III  Patients 

Rahul  Mhaskar,  MPH,  PhDl*,  Branko  Miladinovic,  PhD2*,  Athanasios  Tsalatsanis,  PhD2*, 

Alfred  Mbah,  PhD2*,  Ambuj  Kumar,  MD,  MPH3,  Kim  Sehwan,  PhD4*,  Ronald  Schonwetter, 

MD5*  and  Benjamin  Djulbegovic,  MD,  PhD6 

ICenter  for  Evidence-Based  Medicine,  University  of  South  Florida,  Tampa,  FL;  2USF,  Tampa, 

FL;  3University  of  South  Florida,  College  of  Medicine,  Center  for  Evidence  Based  Medicine, 

Tampa;  4HPC  healthcare,  Tampa,  FL;  5HPC  Healthcare,  Tampa,  FL;  6Center  for  Evidence- 
Based  Medicine  &  Health  Outcomes  Research,  University  of  South  Florida,  Tampa,  FL 

Background:  Over  one  million  Medicare  beneficiaries  receive  hospice  care  annually.  However,  besides 
the  well  documented  advantages  of  hospice,  many  Americans  do  not  enjoy  maximum  benefit  from  the 
hospice  care.  The  fundamental  reason  for  this  is  related  to  the  inappropriate  and  poorly  timed  referral  of 
terminally  ill  patients  to  hospice.  As  a  result,  many  patients  die  within  a  few  days  of  referral,  while  some 
live  many  years  after  the  referral  was  made.  Improvement  in  the  accuracy  of  prognosis  translates  into 
superior  quality  of  care.  Predictions  based  on  statistical  modeling  have  been  shown  to  be  superior  to 
physicians’  prognostication.  However,  very  few  of  these  statistical  models  have  been  externally 
validated  in  terminally  ill  patients.  Here  we  report  the  external  validation  of  5  most  commonly  used 
prognostication  models  in  a  cohort  of  terminally  ill  patients:  1)  declining  exponential  approximation 
of  life  expectancy  (DEALE)  2)  study  to  understand  prognoses  and  preferences  for  outcomes  and  risks 
of  treatments  (SUPPORT),  3)  adjusted  palliative  performance  scale  (PPS),  4)  adjusted  Kamofsky 
performance  scale  index  (Kamofsky)  and  5)  adjusted  eastern  cooperative  oncology  group  performance 
status  (ECOG). 

Methods:  We  retrospectively  extracted  data  from  590  deceased  patients  enrolled  in  Tampa  Bay 
Lifepath  Hospice  and  Palliative  Care  starting  January  2009  and  going  backwards  to  validate  the 
prognostic  models.  Two  research  assistants  extracted  all  data  necessary  to  populate  the  model 
variables  and  two  faculty  members  randomly  checked  25%  of  the  data  for  accuracy.  The  models  were 
tested  against  observed  survival  duration.  PPS,  Kamofsky  and  ECOG  risk  scores  were  predicted  using 
a  flexible  family  of  Royston-Parmar  parametric  models  and  adjusted  for  age,  gender  and  presence  of 
cancer.  We  utilized  several  metrics  to  assess  the  performance  of  these  models.  Specifically,  we  used 
the  Brier  score  and  scaled  Brier  score  (which  is  very  similar  to  the  Pearson  correlation  coefficient  R2), 
the  area  under  the  receiver  operating  characteristic  curve  (AUROC),  and  the  Hosmer-Lemshow 
goodness-of-fit  p-value  (HL). 

Results:  Brier  scores  were  consistently  below  the  non-informative  level  of  0.25  and  AUROC 
significantly  higher  than  the  non-informative  level  of  0.5  for  the  adjusted  PPS,  Kamofsky  and  ECOG 
models  (table  1).  The  HL  p-value  was  consistently  greater  than  0.1  only  for  PPS.  SUPPORT  and 
DEALE  models  did  not  predict  fit  our  data  well  for  survival  at  day  one  and  month  one,  two  and  six. 
The  AUROC  takes  a  value  close  to  0.5,  even  though  the  Brier  scores  were  relatively  low  and  HL  p- 
value  greater  than  0.05,  this  value  is  significantly  close  to  0.5  for  SUPPORT  and  DEALE  models 
(table  1). 
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Conclusion:  None  of  the  prognostication  models  accurately  predicated  survival  among  our  cohort  of 
terminally  ill  patients.  However,  PPS  consistently  performed  best  in  predicting  survival  in  terminally 
ill  patients  followed  by  Kamofsky  and  ECOG. 


Table  1  Discrimination  and  model  performance  statistics  in  survival  prognostication 

Model 

performance 

metrics 

PPS 

Karnofsky 

ECOG 

DEALE 

SUPPORT 

Day  1 

Brier 

0.089 

0.089 

0.106 

0.106 

0.106 

Brier  Scaled 

3.80% 

4.30% 

NA 

NA 

NA 

AUROC  (95%CI) 

0.747  (0.68,  0.813) 

0.747(0.693,  0.810) 

0.709(0.648,  0.771) 

0.526  (0.467,  0.584) 

0.62  (0.56,  0.68) 

HL  p-value 

0.26 

0.17 

0.11 

0.44 

0.58 

Day  3 

Brier 

0.179 

0.178 

0.293 

Brier  Scaled 

16% 

15.30% 

0% 

AUROC 

HL  p-value 

0.768  (0.726,0.810) 
0.29 

0.778(0.737,  0.818) 

0.04 

0.719(0.676,  0.761) 
0.57 

- 

- 

Day  6 

Brier 

0.199 

0.194 

0.363 

(Median) 

Brier  Scaled 

20.10% 

19.40% 

IMA 

AUROC  (95%CI) 

HL  p-value 

0.775  (0.739,0.816) 

0.43 

0.737(0.749,  0.823) 
0.008 

0.721(0.679,  0.764) 

0.44 

- 

- 

Day  10 

Brier 

0.179 

0.183 

0.253 

Brier  Scaled 

26.70% 

25.90% 

4.70% 

AUROC  (95%CI) 

HL  p-value 

0.795  (0.757,  0.834) 
0.15 

0.798(0.761,  0.336) 
0.01 

0.742(0.697,  0.736) 
0.61 

- 

- 

Day  30 

Brier 

0.122 

0.127 

0.095 

0.099 

0.1868 

Brier  Scaled 

37.30% 

37.50% 

12.30% 

2.44% 

11.20% 

AUROC  (95%CI) 

0.731  (0.725,0.833) 

0.737(0.734,0.839) 

0.722(0.651,  0.794) 

0.52  (0.467,  0.573) 

0.56  (0.43,  0.64) 

HL  p-value 

0.62 

0.25 

0.484 

0.92 

0.23 

Day  60 

Brier 

0.084 

0.038 

0.04 

0.045 

0.2 

Brier  Scaled 

47.40% 

47.70% 

18.70% 

16.10% 

13,30% 

AUROC  (95%CI) 

0.745(0.653,  0.837) 

0.781(0.689,  0.371) 

0.739(0.62,  0.358) 

0.543  (0.468,  0.616) 

0.62  (0.5,  0.74) 

HL  p-value 

0.29 

0.32 

0.32 

0.32 

0.67 

Day  180 

Brier 

0.041 

0.05 

0.006 

0.1 

0.27 

Brier  Scaled 

63,20% 

58.60% 

31.20% 

NA 

7.90% 

AUROC  (95%CI) 

HL  p-value 

0.55  (0.452,  0.648) 
0.59 

0.51  (0.314,  0.71) 

0.54 

0.59  (0.355,  0.83) 
0.22 

0.7  (0.336,  1) 

0.58  (0.04, 1) 

0.09 

□:  confidence  Interval,  H-L  Hosmer-Lemshow  statistics,  AU ROC:  area  under  the  receiver  operating  characteristic  curve 
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Rough  Set  Theory  based  Prognostic  Model  for  Hospice  Referral 
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Abstract 

Objective:  The  goal  of  this  paper  is  to  provide  an  accessible  prognostic  classification  model  for 
hospice  referrals.  Hospice  care  provides  high-quality  and  cost-effective  end-of-life  care  for  terminally 
ill  patients.  In  addition,  the  paper  explores  the  application  of  Rough  Set  Theory  (RST)  for  the 
development  of  clinically  credible  prognostic  models. 

Methods:  We  utilize  retrospective  data  from  9,103  terminally  ill  patients  to  demonstrate  the  design 
and  implementation  of  a  classifier  based  on  RST  for  potential  hospice  candidates.  RST  provides 
methods  for  knowledge  reduction,  founded  on  the  relational  indiscemibility  of  objects  in  a  decision 
system,  to  describe  required  conditions  for  membership  in  a  concept  class.  Decision  rules  for  six- 
month  patient  survival  classification  are  extracted  from  the  dataset  utilizing  genetic  algorithms  for 
approximate  reduct  generation. 

Results:  The  RST-based  classifier  performs  comparably  to  other  common  classification  methods, 
while  providing  significant  advantages  in  terms  of  traceability  and  accessibility  of  the  model. 
Conclusions:  In  contrast  to  widely  used  methods  for  prognostic  classification  models,  RST  provides 
several  opportunities  for  adaptation  of  the  model  to  personal  healthcare  preferences.  In  addition,  the 
intuitive  structure  of  the  RST  approach  reflects  the  nature  of  decision-making  and  provides  greater 
insight  into  the  model  process. 

Introduction 

Hospice  care  reduces  the  emotional  burden  of  illness  on  terminal  patients  by  optimizing  pain  relief 
strategies  [1]  and  provides  a  demonstrated,  cost-effective  increase  in  the  quality  of  end-of-life  care 
when  compared  to  conventional  programs  [2],  This  increase  in  quality  of  care  elevates  in  turn  the 
quality  of  life  of  both  patients  and  their  families  [3], 

The  advantages  of  hospice  care  are  diminished  for  terminally  ill  patients  who  enter  either  prematurely 
or  too  late.  In  general,  premature  hospice  referral  represents  a  lost  opportunity  for  the  patient  to 
receive  potentially  effective  and  life-prolonging  treatment.  Conversely,  late  hospice  referral  is  not 
desirable  and  negatively  impacts  both  the  quality  of  end-of-life  care  and  the  quality  of  life  of  patients 
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and  their  families  [4,  5],  According  to  Medicare  regulations,  patient  eligibility  for  hospice  care  is 
determined  by  a  life  expectancy  of  less  than  six  months,  as  estimated  by  the  attending  physician  and 
certified  by  the  medical  director  of  the  hospice  program  [6].  Medicare  claims  data  report  that  14.9%  of 
hospice  care  patients  lived  for  more  than  180  days  after  enrollment,  while  28.5%  were  late  referrals 
who  died  within  14  days  [4,  6].  Accurate  prognostication  of  life  expectancy  is  crucial  in  end-of-life 
care  decisions  and  is  consequently  of  vital  importance  for  patients,  their  physicians  and  their  families. 
Prognostic  models  improve  the  accuracy  of  life  expectancy  estimation  in  critical  clinical  decisions  and 
are  shown  to  be  superior  to  physicians’  prognostication  alone  [7],  Models  for  estimating  the  life 
expectancy  of  terminally  ill  patients  include  the  use  of  traditional  statistical  and  probabilistic  methods 
[8-16]  as  well  as  artificial  intelligence  techniques  such  as  neural  networks  and  support  vector 
machines  [17-19],  decision  trees  [20,  21]  and  rough  set  methods  [22,  23].  The  objective  of  a 
prognostic  model  is  to  determine  a  quantitative  relationship  between  covariates  and  a  health-related 
outcome.  Survival  models  [6,  10,  12,  14,  16,  20,  21]  focus  on  estimating  the  probability  that  a  patient 
will  survive  a  finite  period  of  time.  Other  models,  based  on  neural  networks,  support  vector  machines 
and  logistic  regression  [15,  17-19,  24],  represent  the  survival  outcome  as  a  binary  variable,  predicting 
the  status  of  a  patient  (e.g.  survives/does  not  survive)  at  a  critical  point  in  time  (e.g.  six  months). 

While  prognostic  models  are  an  important  instrument  in  prognostication,  a  significant  barrier  to  their 
widespread  practical  use  is  their  perceived  lack  of  clinical  credibility  [25],  Clinical  credibility  requires 
a  prognostic  model  to  be  both  accurate  and  accessible;  in  addition  to  providing  rigorous,  objective 
information  to  physicians,  the  structure  of  the  model  should  be  transparent  and  its  results  should 
facilitate  physician  and  patient  interpretation.  We  hypothesize  that  accurate  and  accessible  prognostic 
models  based  on  rough  set  theory  can  be  used  to  increase  the  clinical  credibility  and  use  of  prognostic 
models.  To  this  end,  we  will  explore  Rough  Set  Theory  as  it  is  applied  to  end-of-life  care  and  hospice 
referral  decision  support  models. 

This  paper  is  organized  as  follows:  Section  2  presents  a  description  of  important  considerations  when 
analyzing  clinical  data  that  motivate  the  use  of  Rough  Set  Theory  (RST).  In  Section  3,  we  present  an 
overview  of  the  fundamental  theory  of  rough  sets  for  analyzing  datasets.  Section  4  describes  the 
dataset  used  for  the  demonstration  of  the  proposed  prognostic  model.  Section  5  presents  the 
development  of  the  prognostic  model.  Finally,  Sections  6  and  7  reports  the  results,  conclusions,  and 
discusses  limitations  and  future  directions  of  our  work. 

Motivation 

2.1  Inherent  inconsistencies  and  complexity  presented  in  clinical  data 

Clinical  data,  such  as  heart  rate  and  blood  pressure,  represent  the  physiological  state  of  a  patient  at  a 
moment  in  time  and  are  measured  and  recorded  within  a  continuum  of  possible  values.  In  order  to 
facilitate  interpretation,  categorical  variables  representing  each  measured  process  are  defined  along  a 
set  of  cut-points.  For  example,  mean  blood  pressure  can  be  expressed  in  three  discrete  categories: 
high,  normal  and  low  blood  pressure,  divided  by  the  systolic  blood  pressure  cut-points  120  mmHg  and 
90  mmHg.  While  this  representation  supports  rapid  decision  making,  difficulties  arise  when  measured 
values  are  at  or  near  the  cut-points.  In  these  cases,  minor  differences  in  the  value  of  a  measurement 
may  change  the  categorization  of  a  patient,  leading  to  major  changes  in  subsequent  treatment 
decisions  [26].  Moreover,  within  a  dataset  of  numerous  patients,  multiple  patients  may  have  the  same 
categorical  descriptions,  yet  their  outcomes  may  be  varied,  or  inconsistent,  and  as  a  result  the  outcome 
variable  is  vaguely  defined  [27].  A  prognostic  model  that  attempts  to  categorize  patients  as  potentially 
belonging  to  one  of  a  number  of  outcome  groups  must  address  this  vagueness. 

The  presence  of  an  inconsistency  is  often  assumed  to  be  the  result  of  noise  or  an  entry  or  calculation 
error  and  can  thus  be  eliminated  from  the  dataset  or  corrected.  Traditionally,  data  preprocessing 
procedures  address  these  inconsistencies  to  create  consistent  datasets  that  are  then  used  to  train  the 
prognostic  model  and  test  its  performance  [28],  In  clinical  datasets,  however,  inconsistencies  are  the 
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inherent  result  of  the  complex  reality  of  illness  and  human  physiology.  A  prognostic  model  trained 
with  an  inconsistent  dataset  performs  better  at  classifying  an  inconsistent  testing  set  than  a  model 
trained  with  a  consistent  dataset  [29],  and  thus  is  expected  to  perform  better  at  classifying  new  patients 
drawn  from  an  inherently  inconsistent  reality. 

Prognostic  factors,  contained  in  clinical  datasets  of  patients’  test  results  and  physiological 
characteristics,  have  complex  relationships  with  the  outcome  variable.  For  instance,  temperature, 
mean  blood  pressure,  cholesterol  levels,  and  glucose  levels-among  others-all  present  non-linear 
relationships  with  the  survival  time  of  terminally  ill  patients.  Statistical  approaches  make  assumptions 
regarding  the  relationship  between  the  prognostic  factors  and  the  outcome  variable.  When  these 
assumptions  are  violated,  the  resultant  model  is  no  longer  representative  of  the  data.  As  an  example  of 
one  such  assumption,  logistic  regression  assumes  a  linear  relationship  exists  between  a  given 
prognostic  factor  and  the  logit  form  of  the  outcome  variable  [30],  If  the  relationship  is  not  linear  as 
assumed,  the  statistical  significance  of  the  logistic  regression  coefficient  related  to  that  prognostic 
factor  will  be  inaccurate  [31]. 

2.2  Clinical  credibility  and  interpretation  of  prognostic  models  in  clinical  practice 

Prognostic  models,  in  conjunction  with  direct  physician  observation,  increase  the  accuracy  of 
prognostication  when  compared  to  physician  observation  alone  [32],  A  recent  review  [13] 
demonstrated  that,  despite  the  importance  of  accurate  prognostication  within  the  spectrum  of  medical 
care  objectives,  there  is  a  lack  of  accessible  and  accurate  prognostic  models  available  to  physicians  in 
practice. 

To  withstand  clinical  trials,  and  to  meet  the  needs  of  physicians  and  patients,  a  prognostic  model  must 
have  clinical  credibility.  That  is,  however  accurate  or  clinically  effective  a  model  is  statistically, 
physicians  must  believe  in  the  value  of  the  model  as  a  prognostic  tool.  As  such,  a  successful  model 
must  be  accessible,  the  data  required  for  the  model  must  be  relevant  and  simple  to  collect  with  high 
reliability,  the  structure  of  the  model  should  be  apparent  and  its  predictions  should  make  sense  to  the 
physicians  who  rely  on  them.  Physicians  must  be  able  to  apply  the  modeling  method  correctly  without 
violating  the  fundamental  assumptions  of  the  model. 

For  these  reasons,  black  box  models,  such  as  those  based  on  neural  networks  and  support  vector 
machines,  are  less  suitable  as  they  offer  little  insight  into  the  process  of  prediction  and  are  difficult  to 
interpret.  Similarly,  logistic  regression  models  require  careful  application  without  which  results  may 
be  invalidated.  The  interpretation  of  regression  model  results  requires  complex  calculations,  limiting 
the  accessibility  of  the  model.  Additionally,  black  box  models  and  logistic  regression  models  require 
collection  of  data  and  test  results  for  all  of  the  prognostic  factors  used  in  the  model  calibration,  further 
reducing  accessibility. 

2.3  Rough  Set  Theory  in  medical  prognostic  models 

Rough  Set  Theory  [33]  is  a  mathematical  tool  for  data  analysis  that  has  been  used  to  address 
vagueness  and  inconsistencies  present  in  datasets  [34],  In  the  medical  field,  applications  of  RST  focus 
mainly  on  the  diagnosis  and  prognostication  of  diseases,  where  it  has  been  demonstrated  that  RST  is 
very  useful  for  extracting  medical  prognostic  rules  using  minimum  information.  Tsumoto  et.  al.  [35] 
argue  that  the  concepts  of  approximation  established  in  RST  reflect  the  characteristics  of  medical 
reasoning,  explaining  why  RST  performs  well  in  the  medical  field.  For  example,  RST  can  be  used  to 
highlight  non-essential  prognostic  factors  in  a  particular  diagnosis,  thus  helping  to  avoid  redundant, 
superfluous  or  costly  tests  [36-40].  Recently,  methods  that  combine  survival  analysis  techniques  and 
RST  have  been  used  to  generate  prognostic  rules  that  estimate  the  survival  time  of  a  patient  [22,  23], 
RST  provides  a  systematic  approach  for  analyzing  data  without  making  assumptions  common  to 
statistical  methods  (such  as  a  linear  factor-outcome  relationship).  This  advantage  over  statistical 
approaches  makes  RST  suitable  for  integration  into  medical  applications  [41].  The  information 
extracted  from  the  dataset  can  be  represented  in  the  form  of  “if-then”  decision  rules — an  intuitive 
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representation  that  offers  significant  advantage  over  “black  box”  modeling  approaches  [42] — 
increasing  accessibility  and  thus  clinical  credibility. 

Overview  of  Rough  Set  Theory 

Rough  Set  Theory,  introduced  by  Pawlak  in  [33],  provides  methods  for  knowledge  reduction  by 
exploiting  the  relational  indiscemibility  of  objects  in  an  information  system.  Central  to  RST  is  the 
notion  that  an  observed  object  has  a  certain  amount  of  information  associated  with  it.  When 
considered  in  relation  to  a  cohort  of  observed  objects,  this  information  is  used  to  group  similar  objects 
into  information  granules.  Together,  the  information  provided  by  the  set  of  observed  objects  can  be 
generalized  to  describe  the  conditions  required  for  membership  in  a  concept  class. 


Table  1.  Example  Decision  Table 

Condition  Attributes 

Decision  Attribute 

Cl 

Cl 

c3 

c4 

c5 

Ce 

d 

Patient 

Gender 

Age 

SystBP 

HDL  Diabetic 

Smoker 

Coronary  Disease 

x± 

F 

H 

M 

L 

No 

No 

No 

*2 

M 

L 

L 

L 

No 

Yes 

No 

*3 

F 

M 

M 

H 

No 

No 

No 

X4 

F 

M 

M 

H 

No 

No 

Yes 

*5 

M 

H 

H 

L 

Yes 

Yes 

Yes 

X6 

M 

H 

H 

L 

Yes 

Yes 

Yes 

x7 

F 

M 

M 

H 

No 

No 

Yes 

Gender:  Female/Male;  Age:  L  =  [54-59),  M  =  [59-69),  H  =  [69-74];  SystBP  :  L  =  <129,  M 
=  (129-139],  H  =  (139-159];  HDL:  L  =  13,  M  =  (44-60],  H  =  (60-114] 


3.1  Notation 

The  methods  of  RST  act  upon  an  information  system  of  the  form  =  V,f)  ,  where  U  is  a  non¬ 
empty  finite  set  of  objects,  called  the  universe.  A  =  C  U  {d}  is  a  set  of  attributes  that  describe  a  given 
object  in  U,  comprised  of  a  set  C  of  condition  attributes  and  an  optional  decision  attribute  d.  When  d 
is  present,  the  information  system  is  a  decision  system,  and  is  typically  presented  in  table  form.  Given 
an  object  x  E  U,  for  every  attribute  a  E  A,  f(x,  a):  a  -*  Va  maps  the  condition  attribute  of  object  x  to 
its  associated  value  v  E  Va  by  the  information  function  /.  A  value  attribute  pair  (a,  v )  for  a  given 
object  is  referred  to  as  a  descriptor.  The  value  set  of  attribute  a  is  Va,  and  the  set  of  all  values  is  V. 

A  data  requirement  for  RST  is  that  the  attribute  values  must  be  in  discrete  or  categorical  form.  Table  1 
provides  an  example  of  a  discretized  decision  table,  where  six  prognostic  factors,  as  the  condition 
attributes,  describe  seven  patients.  The  decision  attribute,  presence  of  coronary  disease  in  the  patient, 
is  represented  by  the  binary  attribute  d  —>  {Yes,  No}  . 

Once  discretized,  the  objects  in  a  decision  table  can  be  grouped  according  to  their  descriptors.  For 
example,  patients  x5  and  x6  have  the  same  attribute  values  and  are  thus  indiscernible  from  each  other. 
In  general,  two  objects  xt,  Xj  E  U  are  indiscernible  with  respect  to  set  of  condition  attributes  B  Q  A  if 
/(X;,  a)  =  f{xj,  a)  for  all  a  E  B.  This  relation  is  called  an  indiscemibility  relation,  given  by  R(B)  = 
{(xj ,Xj)  E  U:Va  E  B,f{Xi,a )  =  f(Xj,a)}. 

For  example,  the  patients  in  Table  1  can  be  separated  into  four  groups  according  to  the  indiscemibility 
relation  R(C):  X1  =  {x^,  X2  =  {x2},  X3  =  {x3,X4,x7},  X4  =  (x5,x6).  These  groups  of  objects  are 
referred  to  as  equivalence  classes,  or  conditional  classes  for  B  Q  C.  An  equivalence  class  for  the 
decision  attribute  is  called  a  decision  class  or  concept,  and  in  this  example  there  are  two  groups: 
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YNo  =  {x1(  x2,  *3}  and  YYes  =  {x4,  x5>  x6,  x7}.  The  equivalence  class  specified  by  the  object  xt  with 
respect  to  R(B)  is  denoted  as  \x{\B. 

3.2  Set  Approximations 

The  goal  of  RST  is  to  provide  a  definition  of  a  concept  according  to  the  attributes  of  the  equivalence 
classes  that  contain  objects  that  are  known  instantiations  of  the  concept.  As  such,  in  a  consistent 
decision  table,  membership  in  a  conditional  class  implies  membership  in  a  particular  decision  class.  In 
Table  1,  x  E  X4  implies  x  E  YYes.  Membership  in  X3,  however,  does  not  imply  YYes  as  x4,  x7  E  YYes 
but  x3  £YNo.  Thus  Table  1  is  inconsistent  as  d(x4,x7)  =£  d(x3). 

To  represent  an  inconsistent  decision  table,  RST  establishes  an  upper  and  lower  approximation  for 
each  decision  class,  Y.  The  lower  approximation  is  comprised  of  all  objects  that  definitely  belong  to  Y, 
while  the  upper  approximation  includes  all  objects  that  possibly  belong  to  Y .  It  can  be  said  that  an 
object  xt  definitely  belongs  to  a  concept  Y  if  [Xj]c  Q  Y  and  that  xt  possibly  belongs  to  a  concept  Y  if 
[xt\c  n  Y  =£  0.  Thus,  the  lower  and  upper  approximations  are  defined  as  follows: 

RsOO  =  {x  E  U:  [x\B  c  7}  =  \J{[x]b:  [x\b  E  7} 

RbOO  =  {XEU:  [x]b  n  7  *  0}  =  \J{[x]b:  [x]b  n  7  *  0} 

Rb(Y)-Rb(Y)  =  BNDb(Y ) 

The  boundary  region,  BNDb(Y ),  contains  those  objects  that  possibly,  but  not  certainly,  belong  to  7. 
Conversely,  the  set  U  —  RB(Y)  is  the  outside  region  containing  those  objects  that  certainly  do  not 
belong  to  7.  In  our  example,  the  lower  and  upper  approximations  for  YYes  are  R_c(YYes )  =  X4  = 

{x5,  x6}  and  Rc(YYes )  =  X4  U  X3  =  {x3,  x4,  x5,  x6,  x7],  and  the  boundary  region  contains  the  objects 
BNDB(YYes )  =  [x3,  x4,  x7}. 


3.3  Reduct  Generation 


Within  a  decision  system,  not  all  of  the  condition  attributes  may  be  required  to  define  object-concept 
allocation.  If,  for  an  attribute  subset  B  Q  C,  the  indiscemibility  relation  RB  =  Rc,  then  the  set 
approximations  remain  the  same,  the  structure  of  the  decision  system  is  preserved  and  the  attributes  in 
C  —  B  are  said  to  be  dispensable.  There  may  be  many  such  subsets,  but  if  B  is  minimal  (does  not 
contain  any  dispensable  attributes),  then  B  is  termed  a  reduct.  {Age,  Smoker}  and  {SystBP,  HDL}  are 
two  such  reducts  from  our  example  decision  table  that  provide  the  same  quality  of  information  as  the 
complete  set  of  attributes. 

Finding  rough  set  reducts  can  be  framed  as  a  minimal  hitting  sets  problem  [43].  In  this  context,  a 
discemibility  matrix,  MA,  is  constructed  such  that  each  entry  in  MA(xit  xf)  represents  those  attributes 
that  differ,  or  discern,  between  object  xt  and  Xj  with  respect  to  the  attributes  in  A. 


Ma {xi.Xj)  =  {a  G  A:a(Xj)  =£  a(x;)  and  d(Xj)  =£  d(xj)} 

If  a  multiset  C  is  constructed  from  the  non-empty  entries  in  MA,  the  minimal  hitting  sets  are  rough  set 
reducts.  Let  C(S)  be  a  mapping  of  C:  2 u  -»  N  such  that  C(S)  counts  the  number  of  times  a  given  set  S 
appears  in  C,  and  let  /if  — >  {0,1}  be  1  if  A  intersects  a  set  in  C  and  0  otherwise.  Then  af  (X)  is  a 
measure  of  approximation  for  a  set  X  with  respect  to  C 

cm  _  2se2uw(5)C(5)/if(5) 

Ise2"w(S)C(S) 

where  w(S)  provides  a  weight  for  each  set  S.  An  r-approximate  hitting  set  of  C  is  then  a  set  X  Q  U 
such  that  af  (A)  >  r. 

The  computational  cost  for  reduct  computation  is  exponential  with  respect  to  the  size  of  the  decision 
table.  Genetic  algorithms,  operating  based  on  the  principle  of  survival  of  the  fittest,  can  be  used  to 
manage  the  computational  complexity  of  the  dimensionality  reduction  process  [44,  45].  In  this 
process,  an  initial  population  of  elements  is  randomly  selected  from  2A,  and  their  quality  as  a  reduct  is 
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evaluated  using  a  fitness  function.  In  the  case  of  rough  set  reducts,  the  fitness  function  rewards 
elements  that  hit  more  sets  in  C  with  an  approximation  greater  than  r.  A  process  of  combinations, 
mutations  and  random  variations  creates  a  new  generation  of  elements,  rewarding  the  more  fit 
elements  and  discarding  unfit  elements.  The  stopping  criterion  is  a  lack  of  improvement  in  the  average 
fitness  of  the  population  over  a  specified  number  of  generations.  The  result  is  a  collection  of  the  m 
fittest  individuals  in  k  approximation  intervals  between  r  and  1 . 

3.4  Decision  Rules 

The  reduct  generation  process  may  result  in  a  number  of  reducts,  which  may  then  be  filtered 
algorithmically  or  by  the  decision  maker.  Once  the  final  reduct  list  has  been  established,  each  object  in 
the  decision  table  is  used  to  generate  a  rule  based  on  the  attribute  values  of  each  object  for  each  reduct. 
The  collection  of  these  rules  can  then  be  used  to  classify  unseen  objects  -  in  the  case  of  our  example 
table,  a  new  patient  who  may  have  cardiac  disease. 

A  decision  rule  has  the  form:  if  A  then  B  or  A  ->  B,  where  A  is  called  the  antecedent  and  B  the 
consequent  of  the  rule.  Each  object-reduct  pair  contributes  a  rule.  The  antecedent  is  the  logical 
conjunction  of  descriptors  of  the  object  for  the  attributes  in  the  given  reduct  and  the  consequent  is  the 
decision  attribute  of  the  object.  For  example,  object  x1  from  our  example  in  Table  1  yields  the 
following  rule  for  the  reduct  {Age,  Smoker} :  if  Age= H  and  Smoker= No  then  Coronary  Disease  =  No. 

Dataset  description 

The  dataset  used  for  the  development  of  the  prognostic  model  to  assist  in  hospice  referral  consists  of 
9,105  cases  from  the  SUPPORT  (Study  to  Understand  Prognoses  and  Preferences  for  Outcomes  and 
Risks  of  Treatments)  prognostic  model  dataset  [46],  The  SUPPORT  study  included  a  prognostic 
model  for  180-day  survival  estimation  of  seriously  ill  hospitalized  adults  based  on  cubic  splines  and  a 
Cox  regression  model.  As  such,  the  dataset  is  ideal  for  the  present  research  in  regards  to  completeness 
of  data,  comparability  of  results  and  clinical  applicability. 

We  consider  as  condition  attributes  the  variables  used  in  the  SUPPORT  prognostic  model  equation 
[10]  to  ensure  consistency.  The  SUPPORT  variables  include  ten  physiologic  variables  in  addition  to 
the  diagnosis  groups,  age,  number  of  days  in  the  hospital  before  entering  the  study,  presence  of  cancer, 
and  neurologic  function.  Attribute  names,  descriptions  and  value  ranges  are  listed  in  Table  2. 

The  decision  attribute,  a  binary  (Yes/No)  variable,  d.6months  (deceased  in  six  months),  was  defined 
by  comparing  the  values  provided  by  the  SUPPORT  dataset  for  the  variables  death  and  d.time.  The 
former  represents  the  event  of  death  up  to  a  date  12  months  after  the  conclusion  of  the  SUPPORT 
study.  The  latter  records  the  number  of  days  of  follow  up  in  the  study.  If  the  follow  up  time  for  a 
patient  was  less  than  six  months  and  death=Yes,  then  d.6months=Yes.  Otherwise,  it  is  implied  the 
patient  survived  the  6-month  period  and  d.6months=No,  as  study  patients  were  followed  for  a 
minimum  of  180-days. 

Table  2.  Description  of  attributes  from  SUPPORT  dataset 

Numerical  Condition  Attributes  Patient  Distribution 


Variable 


Name 

Description 

Range 

Mean 

Dev. 

age 

Age  of  the  patients 

18-101 

62.65 

15.59 

alb 

Serum  albumin 

0.4  -  29 

2.95 

0.87 

bill 

Bilirubin 

0.1  -63 

2.55 

5.32 

crea 

Serum  creatinine 

0.09-21.5 

1.77 

1.69 

hday 

Number  of  days  in  hospital  at  study  entry 

1  -  148 

1.00 

9.13 

hrt 

Heart  Rate 

0-300 

97.16 

31.56 

meanbp 

Mean  arterial  blood  pressure 

0-195 

84.55 

27.70 

pafi 

Blood  gasses,  PaO2/(.01  *  Fi02) 

12  -  890.4 

239.50 

109.70 

resp 

Respiration  rate 

0-90 

23.33 

9.57 

scoma 

SUPPORT  coma  score,  based  on  Glasgow 
coma  scale 

0-100 

12.06 

24.63 
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sod 

Sodium 

110-181 

137.60 

temp 

Temperature  in  °C 

31.7-41.7 

37.10 

wblc 

White  blood  cell  count 

0-200 

12.35 

Categorical  Condition  Attributes 

Patients 

Percentage 

dzgroup 

Diagnosis  group: 

ARF/MOSF  w  Sepsis 

3,515 

38.60% 

COPD 

967 

10.60% 

CHF 

1,387 

15.23% 

Cirrhosis 

508 

5.56% 

Coma 

596 

6.54% 

Colon  Cancer 

512 

5.62% 

Lung  Cancer 

908 

9.97% 

MOSF  w  Malignancy 

m 

7.81% 

ca 

Presence  of  cancer: 

Yes 

1,252 

13.75% 

No 

5,995 

65.84% 

Metastasis 

1,858 

20.40% 

Decision  Attribute 

Patients 

Percentage 

d.  6months 

Death  occurred  within  6  months: 

Yes 

4,840 

53.17% 

No 

4,263 

46.83% 

While  the  median  survival  time  for  the  patients  in  the  study  is  223  days,  Figure  1  demonstrates  that  the 
majority  of  the  patients  fall  into  one  of  two  extremes:  death  within  40  days  of  study  entry  (2,752 
patients  or  30%  of  all  patients),  or  survival  for  more  than  a  year  (4,000  patients  or  44%). 
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Figure  1.  Distribution  of  patients  with  respect  to  number  of  days  until  death 

General  observations  can  be  made  regarding  the  influence  of  condition  attributes  on  d.  6months  by 
analyzing  the  distribution  of  time  until  death  by  attribute.  For  example,  the  box-whisker  plot  in  Figure 
2  shows  that  a  significant  portion  (75%)  of  patients  with  coma  or  multi-organ  system  failure  with 
malignancy  (MOSF  w/  Malig)  do  not  survive  longer  than  180  days,  but  patients  with  congestive  heart 
failure  (CHF)  or  chronic  obstructive  pulmonary  disease  (CPD)  tend  to  live  longer  than  180  days. 

Thus,  we  can  expect  that  dzgroup  will  be  an  important  attribute  in  the  decision  table. 
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Note,  also,  that  several  dzgroup  categories  have  a  number  of  outliers,  represented  by  circles  in  Figure 
2.  Whereas  the  information  from  these  patients  would  be  lost  in  a  regression  model,  RST-based 
methods  retain  the  information  from  these  patients  in  the  rule-generation  and  -application  process. 
Given  the  number  of  outliers  presented,  however,  it  is  reasonable  to  expect  that  a  reduct 
approximation  scheme  will  be  necessary  to  generate  meaningful  reducts  from  the  decision  table. 
Figure  2.  Survival  time  in  number  of  days  vs.  dzgroup 


ARF/MOSF  w£>epsts  CHF  Crrtnsis  Colon  Cancer  Coma  COPD  Limg  Cancer  MOSF  wtMafeg 


20 


Diagnosis  urouns  tdzerounl 


Development  of  the  prognostic  model 

The  objective  of  the  presented  research  is  to  develop  a  classification  model  for  life-expectancy 
prognostication  using  rough  set  methods.  The  proposed  prognostic  model  was  developed  according  to 
the  following  steps:  data  preprocessing,  discretization,  reduct  generation,  rule  induction  and  rule 
application. 

5.1  Data  preprocessing 

In  its  published  form,  the  SUPPORT  dataset  contains  9,103  complete  cases.  Prior  to  publication,  a 
series  of  analyses  was  done  in  [10]  to  determine  the  most  appropriate  approach  for  imputing  missing 
values  where  needed.  Missing  physiologic  variables  were  then  imputed  with  a  fill-in  value  within  the 
normal  range  of  the  variable.  A  patient  for  whom  it  was  not  possible  to  establish  a  Glasgow  coma 
score  was  given  a  scoma  value  of  zero. 

5.2  Discretization 

As  the  indiscemibility  relations  are  computed  on  categorical  condition  attributes,  RST  methods 
require  that  data  be  categorized.  Discretization  is  the  process  by  which  appropriate  categorical  ranges 
are  found  for  variables  with  a  continuous  value  range. 

There  are  a  number  of  methods  available  for  algorithmic  discretization  that  operate  without  input  from 
the  decision  maker  and  are  based  only  on  the  information  available  in  the  data  table.  In  this  work, 
however,  discretization  was  primarily  performed  using  the  APACHE  III  Scoring  System  [9],  a 
clinically  accepted  scoring  system  designed  to  estimate  the  risk  of  death  in  ICU  patients.  This  choice 
is  founded  on  the  principle  that  medically  and  contextually  relevant  classification  rules  will  increase 
the  clinical  credibility  of  the  proposed  prognostic  model. 

Using  the  cut-points  defined  in  APACHE  III,  nine  physiologic  variables  and  the  age  variable  were 
discretized.  The  physiologic  variables  not  categorized  by  APACHE  III,  pafi  and  scoma,  were 
discretized  using  clinically  accepted  categorizations  [47,  48],  The  variable  hday  was  discretized  using 
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the  Boolean  Reasoning  Algorithm  [49]  as  implemented  in  the  ROSETTA  software  [50],  Table  3 
shows  the  categories  obtained  by  this  process. 

Table  3.  Discretized  attributes  not  in  APACHE  III 


Attribute 

Description 

Categorization 

scoma 

Minor 

(*,9] 

Moderate 

(9,44] 

Severe 

(44,*) 

pafi 

Normal 

[300,*) 

Severe  defect  in  gas  exchange 

[200,300) 

Acute  respiratory  distress  syndrome 

[0,200) 

hday 

Boolean  Reasoning  Algorithm 

(*,44] 

Results 

(44,*) 

Once  discretized,  the  dataset  contains  a  number  of  indiscernible  patients  with  inconsistent  decision 
attributes.  For  example,  the  lower  approximation  of  the  decision  class  d.6months  =  Yes  contains  4,035 
patients  and  the  upper  approximation  contains  4,552  patients,  leaving  517  patients  in  the  boundary 
region. 

At  this  point,  the  dataset  is  randomly  divided  into  training  and  testing  sets  containing  7,282  (80%)  and 
1,821  cases  (20%),  respectively.  The  training  set  is  used  to  develop  the  classification  model  while  the 
testing  set  is  reserved  for  evaluating  the  model’s  performance  when  classifying  unseen  cases. 

5.3  Reduct  Generation 

There  is  exactly  one  minimal  reduct  for  the  SUPPORT  dataset,  which  is  the  set  of  all  condition 
attributes.  Therefore,  it  is  not  possible  to  reduce  the  dimensionality  of  the  decision  table  without 
affecting  the  indiscemibility  relation.  Ziarko  outlined  in  [51]  a  method  to  allow  an  acceptable  level  of 
uncertainty  in  set  approximations,  called  variable  precision  rough  set  method  (VPRS),  which  has  been 
used  in  various  applications  such  as  in  EEG  analysis,  liver  malfunction  diagnosis,  and  corporate 
failure  prediction  [38,  52,  53],  VPRS  allows  for  allocating  equivalence  classes  in  the  boundary  region 
to  the  lower  approximation  of  the  majority  decision  class  for  a  given  certainty  threshold  (0.5  <  /?  < 

1).  This  method  was  explored,  however  the  set  of  all  condition  attributes  was  found  to  be  minimal  for 
all  (3  levels,  indicating  that  reduct  approximation  methods  would  be  more  appropriate  for  rule 
induction. 

Approximate  reducts  were  obtained  by  applying  the  minimal  hitting  set  method  discussed  in  Section 

3.3  using  the  genetic  algorithm  implemented  in  ROSETTA  [50],  at  varying  levels  of  approximation, 
as  shown  in  Table  4.  As  an  indicator  of  the  level  of  importance  of  an  attribute,  the  number  of  reducts 
in  which  an  attribute  appears  is  counted  and  used  to  rank  each  attribute.  The  sum  of  rankings  across  all 
levels  of  r  indicates  the  overall  importance  of  each  attribute.  It  can  be  seen  that  dzgroup,  meanbp,  and 
age  are  the  most  important  attributes  for  the  180-day  survival  classifier. 


Table  4.  Condition  attributes  ranked  by  appearance  in  reducts. 

r  =  0.1  r=  0.2  r  =  0.4  r  =  0.6  r=0.8  Rank 

Attributes  %  Rank  %  Rank  %  Rank  %  Rank  %  Rank  Score 

dzgroup  74.1%  3  78.4%  2  81.7%  3  94.8%  1  97.1%  1  10 

meanbp  74.1%  2  77.8%  3  88.6%  1  92.9%  2  94.7%  2  10 
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age 

74.6% 

1 

79.4% 

1 

88.0% 

2 

87.7% 

3 

92.6% 

4 

11 

resp 

58.4% 

5 

68.6% 

4 

78.3% 

4 

79.4% 

4 

89.5% 

5 

22 

hrt 

61.4% 

4 

65.0% 

5 

65.1% 

6 

79.4% 

5 

94.7% 

3 

23 

pafi 

49.8% 

6 

55.2% 

6 

72.0% 

5 

78.1% 

6 

86.3% 

6 

29 

crea 

39.6% 

8 

48.5% 

7 

58.9% 

7 

67.1% 

7 

85.3% 

7 

36 

scoma 

41.6% 

7 

46.4% 

8 

58.9% 

8 

61.9% 

9 

79.0% 

8 

40 

sod 

33.0% 

10 

42.3% 

9 

43.4% 

10 

52.3% 

12 

73.7% 

9 

50 

alb 

26.2% 

12 

28.4% 

11 

34.9% 

13 

66.5% 

8 

68.4% 

11 

55 

ca 

25.4% 

13 

18.6% 

14 

44.6% 

9 

56.1% 

10 

69.5% 

10 

56 

bili 

26.4% 

11 

24.7% 

12 

40.6% 

11 

54.8% 

11 

66.3% 

12 

57 

wblc 

35.0% 

9 

35.6% 

10 

39.4% 

12 

42.6% 

13 

62.1% 

14 

58 

temp 

18.8% 

14 

19.1% 

13 

30.3% 

14 

38.7% 

14 

66.3% 

13 

68 

hday 

7.1% 

15 

9.8% 

15 

22.9% 

15 

15.5% 

15 

13.7% 

15 

75 

Total  197  reducts  194  reducts  175  reducts  155  reducts  95  reducts 


5.4  Rule  Induction 


For  each  r-approximate  reduct  set,  every  patient  in  the  training  set  contributes  to  a  decision  rule  of  the 
form  if  A  ->  B  based  on  the  reduced  decision  table  for  each  reduct.  For  each  rule,  the  left  hand  side 
(LHS)  support  is  the  number  of  patients  in  the  table  whose  attributes  match  the  antecedent,  A,  while 
the  right  hand  side  (RHS)  support  indicates  the  number  of  patients  matching  the  consequent  of  the 
rule,  B.  The  number  and  selection  of  descriptors  in  the  antecedent  varies  according  to  the  reduct 
generating  the  rule,  thus  the  set  of  all  r-approximate  decision  rules  obtained  from  the  training  set  will 
contain  rules  of  varying  number  of  descriptors. 

Several  decision  rules  are  highlighted  in  Table  5.  Rules  1  and  2  contain  only  two  descriptors  and 
therefore  match  more  patients  in  the  training  set  leading  to  a  larger  LHS  support.  Note  that  these  rules 
match  the  expectations  from  our  general  observations  from  Section  4  about  patients  with  CHF  or 
MOSF  with  malignancy.  Rules  3  and  4  contain  six  and  four  descriptors  respectively,  and  are  thus 
more  specific. 


Table  5.  Selected  decision  rules  for  r  =  0.1  approximate  reducts. 


_ Rule _ LHS 

1.  If  dzgroup=CHF  AND  age=(74,84]  254 

— »  d.6months={ Yes, No} 

2.  //c/zg/'ou/?=MOSFw/Malig  AND  hrt=(49,99]  192 

— »  d.6months= {Yes, No} 

3.  If  dzgroup=CHF  AND  age=(74,84]  AND  8 

resp=(  13,24]  AND  alb=(2.4,4.4]  AND 

crea=(  1.4,1.94]  AND  sod=(  1 34, 1 54] 

— »  d.6months= {Yes ,No} 

4.  7Fdzgrowp=MOSFw/Malig  AND  hrt={ 49,99]  7 

AND  meanbp=( 99,1 19]  AND  resp=( 5,1 1] 

— >  d.6months={ Yes, No} 


_ RHS  Support _ 

d.6months  =  d.6months  = 
Yes  No 


73  (29%) 

181  (71%) 

128  (67%) 

64  (33%) 

6  (75%) 

2  (25%) 

1  (14%) 

6  (86%) 
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5.5  Rule  Application 

A  standard  voting  process  [49]  is  used  to  allow  all  rules  to  participate  in  the  decision  process,  arriving 
at  a  patient  classification  by  majority  vote.  Thus,  for  a  new,  unseen  patient,  any  rule  whose  antecedent 
descriptors  match  the  patient  descriptors  “fires”  by  contributing  as  votes  the  RHS  support  for  each 
decision  class.  Once  all  rules  have  “voted”,  the  number  of  votes  for  each  decision  class  is  normalized 
against  the  total  number  of  LHS  support  for  all  fired  rules.  The  resultant  ratio  of  RHS  to  LHS  support 
is  considered  a  frequency-based  estimate  of  the  probability  that  the  patient  belongs  to  the  given 
decision  class. 

As  an  example,  consider  a  patient  whose  profile  includes  the  following  descriptors:  dzgroup=CW, 
age={ 74,84],  resp={  13,24],  alb=(2.4,4.4]  ,  crca={  1 .4. 1 .94  ].  and  sod={  134,154],  If  the  prognostic 
model  were  applied  to  this  patient,  then  both  rule  1  and  3  would  fire.  Supposing  rule  1  and  3  are  the 
only  rules  that  fire  for  this  example  patient,  the  probability  that  the  patient  will  survive  six  months  is 
estimated  as  (181  +  2)/(254  +  8)  =  69.8%.  Any  additional  rules  matching  the  patient’s  profile 
would  contribute  similarly  to  the  estimated  six-month  survival  probability. 

A  final  classification  is  determined  according  to  a  threshold  value,  r  E  [0,1].  A  patient  is  classified  as 
not  surviving  six  months  if  the  estimated  probability  of  death  in  six  months  is  greater  than  r.  In  the 
event  of  an  estimated  probability  equal  to  r,  or  in  the  absence  of  any  fired  rules  (no  rule  matches  the 
patient  profile),  classification  is  not  possible  and  the  patient  is  labeled  undefined. 

Results 

The  performance  of  the  model  was  tested  by  measuring  the  discriminatory  power  of  each  of  the  r- 
approximate  sets  of  decision  rules  when  applied  to  the  reserved  testing  set.  For  our  notation,  a 
classification  of  d.6months=  Yes  is  referred  to  as  a  positive  classification,  and  d.6months=  No  is 
negative.  Sensitivity  is  defined  as  the  fraction  of  patients  who  did  not  survive  six  months  and  are 
correctly  classified  by  the  model,  or  the  fraction  of  true  positive  classifications  of  all  test  patients  who 
did  not  survive  six  months.  Conversely,  specificity  is  defined  as  the  fraction  of  patients  who  did 
survive  six  months  and  were  correctly  classified  by  the  model,  or  the  fraction  of  true  negatives  of  all 
test  patients  who  did  survive  six  months. 

The  overall  accuracy  of  the  classification  model  is  reported  in  terms  of  area  under  the  Receiver 
Operating  Characteristic  (ROC)  curve,  or  AUC.  The  ROC  curve  graphs  the  sensitivity  of  the  classifier, 
or  the  true  positive  rate,  versus  1 -specificity,  the  false  positive  rate,  as  the  threshold  probability,  r,  for 
positive  classification  is  varied  from  0  to  1 .  The  best  overall  classification  performance  is  realized 
when  AUC  is  equal  to  1,  while  an  AUC  of  0.5  indicates  a  classifier  performance  no  better  than 
random  selection.  Best  separation  between  decision  classes  is  realized  at  the  threshold  corresponding 
to  the  point  on  the  ROC  curve  closest  to  the  point  (0,1). 

Also  of  importance  is  the  coverage  of  the  classification  model,  i.e.  the  percentage  of  testing  set 
patients  for  whom  a  classification  is  possible.  Coverage  and  AUC  for  each  evaluated  /"-approximate 
reduct  are  shown  in  Table  6,  and  the  associated  ROC  curve  for  each  is  illustrated  in  Figure  3.  Clearly, 
r  is  a  critical  value  in  determining  AUC,  coverage  and  maximum  sensitivity  and  specificity  values. 
Classifier  performance  at  a  particular  value  of  r  is  dataset-dependent,  however,  in  general,  r=l 
represents  a  reduct  generation  approach  strictly  aimed  at  dimensionality  reduction,  while  r  closer  to 
zero  represents  a  more  relaxed  criterion  for  reduct  acceptability. 
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Table  6.  AUC  and  coverage  for  various  levels  of  r-approximate  reducts. 


r 

AUC  (%) 

Coverage  (%) 

0.1 

70.3 

100 

0.2 

69.7 

100 

0.4 

69.5 

100 

0.6 

65.9 

98.7 

0.8 

61.3 

66.9 

In  order  to  estimate  the  stability  of  the  classification  model,  a  k-fold  cross  validation  [54]  procedure 
was  applied.  The  entire  dataset  was  randomly  divided  into  five  subsets,  or  folds,  and  then  each  fold 
(20%  of  the  dataset)  is  used  once  as  a  testing  set,  with  the  remaining  folds  (80%)  used  for  training. 

The  results  from  each  of  the  five  fold  configurations  are  averaged  to  provide  an  estimate  for  the 
classifier  performance.  Table  7  presents  the  results  for  the  classification  model  for  each  fold 
arrangement  for  the  three  r-approximation  levels  with  the  highest  AUC,  r=0.1,  0.2,  and  0.4.  The  AUC, 
sensitivity,  and  specificity  for  each  fold  are  shown. 

Table  7.  K-fold  validation  results  for  r-approximate  reducts,  by  reserved  testing  fold. 


r  =  0.1 

^2 

II 

o 

r  =  0.4 

Testing 

AUC 

Sens. 

Spec. 

AUC 

Sens. 

Spec. 

AUC 

Sens. 

Spec. 

Fold 

(%) 

(%) 

(%) 

(%) 

(%) 

(%) 

(%) 

(%) 

(%) 

1 

70.34 

65.28 

66.22 

69.74 

62.78 

67.04 

69.53 

62.78 

67.96 

2 

71.78 

58.94 

74.56 

68.47 

62.59 

63.34 

68.12 

60.35 

67.97 

3 

71.91 

68.74 

64.90 

71.99 

61.93 

69.48 

71.23 

67.90 

65.51 

4 

68.37 

66.86 

61.08 

68.97 

60.93 

68.03 

67.77 

63.66 

63.64 

5 

70.18 

61.77 

68.93 

70.28 

64.57 

66.24 

71.02 

67.49 

65.26 

Mean 

70.52 

64.32 

67.14 

68.89 

62.56 

66.83 

69.53 

64.44 

66.07 

Std.  Dev. 

1.44 

3.95 

5.02 

1.34 

1.34 

2.29 

1.60 

3.22 

1.87 
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6.4  Performance  comparison  with  logistic  regression  and  support  vector  machines 

RST  approaches  in  other  diagnostic  prognostic  models  have  resulted  in  significantly  higher 
discriminatory  performance,  with  typical  AUC  near  90%  [37,  43,  55].  To  ascertain  if  the  performance 
demonstrated  by  the  present  application  is  indicative  of  the  unsuitability  of  RST  for  the  problem  or  of 
the  complexity  of  the  dataset,  the  results  are  compared  with  two  popular  classification  approaches: 
logistic  regression  and  SVM.  Table  8  shows  the  performance  of  the  classification  models,  using  the  5- 
fold  cross  validation  scheme,  in  terms  of  sensitivity,  specificity  and  AUC.  All  three  models  perform 
similarly,  and  an  AUC  of  less  than  75%  for  all  three  highlights  the  complexity  of  the  estimation  of  life 
expectancy  via  a  classification  model. 

Table  8.  Performance  of  three  classification  models:  Logistic  regression,  SVM  and  RST. 


Method 

Sens. 

Std.  Dev 

Spec. 

Std.  Dev. 

AUC 

Std.  Dev. 

Logistic  regression 

59.20% 

0.02 

76.40% 

0.01 

74.10% 

0.99 

Support  vector  machines 

65.52% 

0.02 

70.40% 

0.01 

73.40% 

1.20 

RST 

64.32% 

3.95 

67.14% 

5.02 

70.52% 

1.44 

Discussion 

7.1  Interpretation  and  usability  of  decision  rules 

Clinical  credibility  in  prognostic  models  depends  in  part  on  the  ease  with  which  physicians  and  patients 
can  understand  and  interpret  the  results  of  the  models,  in  addition  to  the  accuracy  of  the  information 
they  provide.  While  the  performance  of  the  RST-based  prognostic  model  is  comparable  to  similar 
methods,  the  if-then  decision  rule  approach  offers  significant  advantages  by  increasing  both  the 
traceability  of  the  model  and  the  amount  of  information  included  in  its  results. 

Consider  an  example  patient  with  MOSF  with  malignancy  who  is  classified  as  not  surviving  six 
months  with  a  probability  of  66%.  Among  the  list  of  fired  rules  presented  to  the  physician  are  rules  2 
and  4  from  Table  5.  Rule  2  indicates  that  of  192  patients  with  dzgroup= MOSF  w/  Malig  and 
heartrate=( 49,99],  128  (67%)  indeed  did  not  survive  six  months.  However,  rule  4  shows  more 
specifically  that  the  example  patient  matches  7  patients  when  two  additional  physical  conditions  are 
considered,  of  whom  6  (86%),  in  fact,  survived  more  than  six  months. 

Thus,  the  gestalt  survival  expectation  is  presented  without  loss  of  contradictory  information.  This 
increases  the  transparency  and  traceability  of  the  classification  process,  strengthening  the  accessibility, 
and  hence  credibility,  of  the  model.  This  is  in  contrast  to  SVM,  neural  networks  and  other  black-box 
methods,  where  the  information  provided  is  limited  to  a  classification  and  a  confidence  interval.  None 
of  these  methods  provide  traceable  and  accessible  results  as  in  the  case  of  the  if-then  decision  rule 
representation. 

Additionally,  logistic  regression  and  other  black-box  models  require  the  complete  set  of  condition 
attributes,  or  prognostic  factors,  as  those  used  in  the  formation  of  the  respective  model.  Contrarily,  the 
RST  approach  also  offers  the  advantage  of  acceptable  results  should  a  particular  prognostic  factor  be 
difficult  or  too  costly  to  ascertain  for  a  patient  [36]. 

7.2  Decision  analysis  for  hospice  referral 

Consider  the  costs  -  economic,  emotional  and  physical  -  associated  with  the  decision  to  enter  hospice 
care.  These  costs  are  justified  for  patients  who  either  enter  hospice  care  at  the  appropriate  time  or  for 
those  who  do  not  enter  hospice  care  when  they  could  benefit  from  curative  treatment.  These  cases 
represent  true  positive  and  true  negative  classifications.  A  higher  emotional  and  physical  cost  is  bom 
by  patients  sent  to  hospice  care  but  who  ultimately  survive  six  months  -  a  false  positive.  The  highest 
cost  of  all,  emotionally,  economically  and  physically  is  bom  by  the  patient  and  his  or  her  family  when 
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costly  treatment  is  prolonged  for  a  patient  who  does  not  survive  six  months  and  who  should  have  been 
referred  to  a  hospice  care  program  -  a  false  negative.  In  this  last  case,  some  or  all  of  the  benefits  of 
hospice  care  would  be  lost  while  the  stresses  and  economic  burden  of  aggressive  treatment  are 
endured. 

In  this  light,  the  threshold  parameter,  r  (described  in  Section  5.5),  can  be  seen  as  a  representation  of 
the  patient  and  family’s  preference  for  hospice  care  treatment  and  their  risk  tolerance  for  a  mistaken 
referral.  The  threshold  parameter  relates  sensitivity  to  specificity  and  stipulates  the  required  level  of 
certainty  for  a  positive  classification.  A  higher  threshold  value  requires  a  higher  probability  of  not 
surviving  six  months  for  the  classification  of  a  patient  as  a  hospice  candidate,  decreasing  the 
sensitivity  and  increasing  specificity  (indicating  a  preference  for  continued  treatment).  Conversely,  a 
lower  threshold  value  increases  sensitivity  while  reducing  specificity,  indicating  a  preference  for 
avoiding  the  costly  mistake  of  unnecessary  treatment. 

As  this  threshold  value  is  a  subjective  matter  and  varies  between  physicians,  patients  and  family 
members,  one  suggested  approach  [56]  involves  the  measurement  of  the  amount  of  regret  the  decision 
maker  would  have  should  an  incorrect  decision  be  made.  As  medical  decisions  must  take  into  account 
the  preferences  of  those  ultimately  affected  by  the  decision,  this  application  of  regret  theory  allows  for 
the  formal  treatment  of  those  preferences  by  calculating  the  threshold  value  as  a  function  of  the 
measured  anticipated  regret. 

Conclusions 

Logistic  regression  models  dominate  clinical  prognostic  models,  largely  as  a  result  of  the  simplicity  of 
its  application  in  widely  available  software  packages,  the  history  of  usage  with  successful  results  in 
the  field,  and  the  ability  to  statistically  interpret  model  parameters  [57],  Logistic  regression,  however, 
is  not  able  to  identify  non-linear  structures  in  the  dataset,  and  its  results  may  be  invalidated  when  its 
assumptions  are  not  met  in  practice. 

So-called  black-box  models  have  seen  increased  usage,  with  generally  improved  accuracy  at  the  cost 
of  drawbacks  in  the  interpretability  of  the  parameters  and  in  the  traceability  of  the  model  process. 
These  drawbacks  have  limited  the  popularity  of  black-box  models  outside  of  domains  in  which 
accuracy  is  of  much  higher  importance  than  interpretability. 

This  paper  contributes  to  the  growing  body  of  research  in  RST  as  a  prognostic  modeling  tool  [37,  39- 
41]  and  highlights  the  strengths  of  this  approach  in  terms  of  accessibility.  RST  is  found  to  perform 
similarly  to  two  common  classification  approaches,  LR  and  SVM,  while  also  offering  more 
information.  The  intuitive  structure  of  the  RST  approach,  built  on  similarity  relations  and  expressed  in 
terms  of  if-then  decision  rules,  offers  both  more  insight  into  the  model  process  and  more  opportunity 
for  the  knowledge  extraction  process  to  incorporate  the  personal  preferences  of  those  making  and 
being  affected  by  the  decision. 

There  is  an  emergent  trend  towards  personalized  healthcare  [58-60]  -  concurrent  with  advances  in  the 
field  of  data  acquisition  and  storage  -  propelled  by  increasingly  available  personal  healthcare  data.  In 
this  respect,  RST  shows  promising  potential  for  data-driven  prognostic  models  in  medicine  that  are 
adaptive  to  personal  healthcare  preferences. 

However,  the  performance  of  classification  models  is  still  a  major  issue  for  the  targeted  domain  of  life 
expectancy  prognostication.  Classifier  performance,  measured  by  AUC,  is  good  but  sub-optimal, 
which  indicates  a  challenging  problem  in  need  of  further  research.  The  increased  performance 
achieved  by  the  reduct  approximation  approach  indicates  a  dataset  of  highly  diverse  patients.  Future 
research  will  explore  methods  to  produce  more  accurate  results  by  addressing  this  diversity. 
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Abstract 

Background:  Despite  the  well  documented  advantages  of  hospice  care,  most  terminally  ill  patients  do  not  reap 
the  maximum  benefit  from  hospice  services,  with  the  majority  of  them  receiving  hospice  care  either  prematurely 
or  delayed.  Decision  systems  to  improve  the  hospice  referral  process  are  sorely  needed. 

Methods:  We  present  a  novel  theoretical  framework  that  is  based  on  well-established  methodologies  of 
prognostication  and  decision  analysis  to  assist  with  the  hospice  referral  process  for  terminally  ill  patients.  We  linked 
the  SUPPORT  statistical  model,  widely  regarded  as  one  of  the  most  accurate  models  for  prognostication  of 
terminally  ill  patients,  with  the  recently  developed  regret  based  decision  curve  analysis  (regret  DCA).  We  extend  the 
regret  DCA  methodology  to  consider  harms  associated  with  the  prognostication  test  as  well  as  harms  and  effects 
of  the  management  strategies.  In  order  to  enable  patients  and  physicians  in  making  these  complex  decisions  in 
real-time,  we  developed  an  easily  accessible  web-based  decision  support  system  available  at  the  point  of  care. 

Results:  The  web-based  decision  support  system  facilitates  the  hospice  referral  process  in  three  steps.  First,  the 
patient  or  surrogate  is  interviewed  to  elicit  his/her  personal  preferences  regarding  the  continuation  of  life- 
sustaining  treatment  vs.  palliative  care.  Then,  regret  DCA  is  employed  to  identify  the  best  strategy  for  the  particular 
patient  in  terms  of  threshold  probability  at  which  he/she  is  indifferent  between  continuation  of  treatment  and  of 
hospice  referral.  Finally,  if  necessary,  the  probabilities  of  survival  and  death  for  the  particular  patient  are  computed 
based  on  the  SUPPORT  prognostication  model  and  contrasted  with  the  patient’s  threshold  probability.  The  web- 
based  design  of  the  CDSS  enables  patients,  physicians,  and  family  members  to  participate  in  the  decision  process 
from  anywhere  internet  access  is  available. 

Conclusions:  We  present  a  theoretical  framework  to  facilitate  the  hospice  referral  process.  Further  rigorous  clinical 
evaluation  including  testing  in  a  prospective  randomized  controlled  trial  is  required  and  planned. 


Background 

Introduction 

Hospice  services  have  been  proven  to  provide  better 
quality  of  care  to  dying  patients[l-3]  by  optimizing  pain 
relief  [4,5]  and  reducing  emotional  stress  [1,6,7]. 
Furthermore,  hospice  care  is  associated  with  greater 
patient-family  satisfaction^],  is  shown  to  be  cost  effec¬ 
tive^,  10],  and  most  importantly,  it  has  been  attributed 
with  increased  survival  in  some  patients  [11].  Despite 
these  well  documented  advantages,  many  terminally  ill 
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patients  do  not  reap  maximum  benefits  from  hospice 
care.  The  fundamental  reason  for  this  is  related  to  the 
less  than  optimal  and  frequently  poorly  timed  refer¬ 
ral  of  terminally  ill  patients  to  hospice  [1,12].  As  a 
result,  many  patients  die  within  a  few  days  of  referral, 
or  live  many  years  after  the  referral  was  made  [13]. 

According  to  Medicare  regulations,  a  person  should  be 
referred  to  hospice  if  his/her  “life  expectancy  (LE)  is  6 
months  or  less”  [1,14].  Hence,  the  problem  of  meaning¬ 
ful  referrals  relates  to  the  accurate  estimation  (prog¬ 
nosis)  of  death  within  approximately  6  months  after 
evaluation  for  hospice  care.  However,  statistical  models 
designed  to  assist  physicians  in  predicting  life 
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expectancy  (LE),  although  beneficial  [15,16],  so  far  they 
failed  to  improve  the  quality  of  care  at  the  end  of  life 
[17-21]. 

One  such  statistical  model  is  SUPPORT  (Study  to 
Understand  Prognoses  and  Preferences  for  Outcomes 
and  Risks  of  Treatments),  designed  to  calculate  the 
probability  of  survival  over  a  period  of  180  days  [22,23]. 
Although  the  SUPPORT  model  has  been  well  validated 
[17,22]  for  prognostication  of  LE  in  terminally  ill 
patients,  a  controlled  trial  of  SUPPORT  failed  to 
demonstrate  any  impact  on  the  overall  quality  of  care 
for  these  patients  [17,20].  We  postulate  that  this  lack  of 
impact  may  be  due  to  the  fact  that  SUPPORT  results, 
were  not  linked  to  any  decision  methodology  that  would 
translate  the  probability  of  survival  to  a  hospice  referral 
recommendation.  Therefore,  the  full  potential  of  the 
model’s  prognostication  power  remained  unexploited. 

In  this  work,  we  link  the  SUPPORT  prognostication 
model  with  the  recently  developed  decision  methodology 
regret  DCA  [24]  to  facilitate  the  hospice  referral  process. 
Regret  DCA  relies  on  regret  theory  and  decision  curve  ana¬ 
lysis  [25]  to  recommend  the  optimal  management  strategy 
for  a  patient,  accounting  for  the  personal  attitudes  and 
values  of  the  particular  patient  or  his/her  surrogate. 

Furthermore,  we  extend  regret  DCA  to  incorporate 
harms  and  effects  of  treatment  as  well  as  harms  asso¬ 
ciated  with  the  prognostication  test  to  the  decision 
model.  The  presented  methodology  is  integrated  into  a 
comprehensive  clinical  decision  support  system  devel¬ 
oped  to  facilitate  the  hospice  referral  process. 

Methods 

Dataset 

In  our  analysis,  we  utilized  the  entire  SUPPORT  dataset, 
both  development  and  validation  cohorts.  The  dataset  is 
presented  in  detail  elsewhere  [22].  Medical  records  of 
8,329  seriously  ill  hospitalized  adults  are  included. 


Support  model 

SUPPORT  is  a  multivariable  model  designed  to  estimate 
probability  of  survival  for  seriously  ill  hospitalized 
patients  over  a  period  of  the  subsequent  180  days.  The 
model  variables  include  the  patient’s  medical  condition 
compatible  with  one  of  eight  major  diagnostic  groupings 
(Acute  Respiratory  Failure,  Multiple  Organ  System  Fail¬ 
ure,  Chronic  Obstructive  Pulmonary  Disease,  Congestive 
Heart  Failure,  Hepatic  Cirrhosis,  Neurological  Coma, 
Lung  or  Colon  Cancer),  the  patient’s  current  age,  num¬ 
ber  of  days  in  the  hospital  before  study  entry,  neurologic 
status,  and  11  physiologic  measures  recorded  on  day  3 
after  study  entry  [22]. 

The  SUPPORT  implementation  for  the  estimation  of 
survival  probability  is  detailed  in  the  appendix.  Due  to 
the  nature  of  the  hospice  referral  problem  we  also 
express  the  survival  probability  in  terms  of  mortality. 
We  can  convert  the  estimated  survival  probability  (SP) 
(equation  A2)  to  probability  of  death  within  180  days 
(denoted  here  as  p)  using  the  equation: 

p  =  1  —  SP  =  \  —  P{T  >  /jdisease  group  -  /}  (1) 

where  SP  is  the  survival  probability  computed  by  SUP¬ 
PORT,  i  I  [1,8]  the  patient’s  disease  group,  T  is  the  sur¬ 
vival  time  in  days,  and  t  is  an  arbitrary  time  (typically 
expressed  in  days  e.g.  t  I  [1,180]). 

In  terms  of  accuracy,  the  SUPPORT  model  has  an 
area  under  the  receiver-operating  characteristics  curve 
(ROC)  for  prediction  of  surviving  180  days  of  0.79  in 
the  phase  I  development  cohort  and  0.78  in  the  phase  II 
validation  cohort  [22]. 

Decision  model 

Figure  1  depicts  the  decision  tree  summarizing  the  pro¬ 
cess  of  hospice  referral.  The  four  outcomes  and  their 
corresponding  utilities  (U)  shown  are: 


LE<6  months 


Refer  to 


Outcomes  Regret  Utilities 

-OHosp|LE<6  Rgi=0  U:  TP 


hospice 


-O 


LE  >  6  months 


i-p 

LE<6  months 


Continue 


“C Z)  Hosp|LE  >6  Rg2=U4_U2  U2  FP 


-O  Rx  |  LE  <6  Rg3=Ui-U3  U3  FN 


Treatment 


-0 


LE  >  6  months 


I-P 


-O  Rx  |  LE  >  6  Rg4=0  U4  TN 


Figure  1  Decision  tree  for  hospice  referral.  In  this  figure,  p  is  the  probability  that  a  patient’s  LE  is  less  than  or  equal  to  6  months;  1-pis  the 
probability  that  a  patient’s  LE  is  greater  than  6  months;  \J{.  are  the  utilities  associated  with  each  outcome;  Rgi  is  the  regret  associated  with  each 
outcome. 

^  j 
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1 .  U i :  Refer  the  patient  to  hospice  and  the  patient’s 
LE  is  less  than  or  equal  to  6  months  (Hosp|LE  <  6). 

2.  U2:  Refer  the  patient  to  hospice  and  the  patient’s 
LE  is  greater  than  6  months  (Hosp|LE  >  6). 

3.  U3:  Continue  treating  the  patient  and  the  patient’s 
LE  is  less  than  or  equal  to  6  months  (Rx|LE  <  6). 

4.  U4:  Continue  treating  the  patient  and  the  patient’s 
LE  is  greater  than  6  months  (Rx|LE  >  6). 


p  is  the  probability  associated  with  the  presence  of  an 
event  (e.g.  patient’s  LE  <  6  months)  as  predicted  by  the 
SUPPORT  model,  1  -  p  is  the  probability  associated 
with  the  absence  of  the  same  event  (e.g.  patient’s  LE  >  6 
months). 

As  with  any  decision,  one  may  come  to  realize  that,  in 
retrospect,  an  alternative  decision  would  have  been  pre¬ 
ferable.  This  knowledge  may  bring  a  sense  of  loss  or 
regret  [26-32].  In  this  paper,  we  use  this  sense  of  regret 
to  determine  the  preferences  of  the  decision  maker 
towards  alternative  management  strategies.  Specifically, 
we  employ  regret  theory  to  estimate  the  threshold  prob¬ 
ability,  Pt,  at  which  the  decision  maker  (patient,  physi¬ 
cian,  or  family  member)  is  indifferent  between 
continuation  of  treatment  vs.  hospice  referral.  Based  on 
the  concept  of  threshold  probability,  the  patient  should 
be  referred  to  hospice  if  his/her  probability  of  death  is 
greater  than  or  equal  to  Pt(e.g.  p  ^  Pt),  and  he/she 
should  continue  receiving  curative  treatment  otherwise 
(p  <Pt). 

The  threshold  probability  is  derived  as  [24]: 


Pt  = 


1 


i  +  Ui-u* 
1  ^  u4-u2 


(2) 


In  (2)  UrU3  is  associated  with  regret  of  omission  (e.g. 
the  patient  was  not  referred  to  hospice,  instead  he/she 
continued  receiving  unnecessary  treatment)  and  U4-U2 
with  regret  of  commission  (e.g.  the  patient  was  unneces¬ 
sary  referred  to  hospice  instead  of  continue  receiving 
life-sustaining  treatment)  [24]. 

To  elicit  the  decision  maker’s  regret,  and  therefore 
threshold  probability,  we  utilize  the  DVAS  (Dual  Visual 
Analogue  Scale)  method  [24].  One  visual  analogue  scale 
is  used  to  capture  the  regret  associated  with  failing  to 
refer  the  patient  to  hospice  (e.g.  continue  unnecessary 
treatment)  and  the  second  scale  to  measure  the  regret 
associated  with  unnecessary  hospice  referral  (e.g.  failing 
to  provide  life- sustaining  treatment)  (Figure  2). 

Elicitation  of  threshold  probability  can  be  achieved 
through  a  set  of  questions  such  as: 


1.  On  the  scale  0  to  100,  where  0  indicates  no  regret 
and  100  indicates  the  maximum  regret  you  could 
feel,  how  would  you  weigh  the  level  of  your  regret  if 


On  0  scale  0  to  100,  where  0  indicates  no  regret  and  100  indicates  the 
maximum  regret  you  could  feel,  how  would  you  rote  the  level  of  your 
regret  if  you  were  not  referred  to  hospice  but  instead  you  continued 
receiving  unnecessary  treatment? 


Nn 

regret 


50 


Maximum 

regret 

>100 


THi£  VAkis  rgprfrMfrhls  !h& consequences  pi  feHure  IQ  refer  IQ  Inspire 

^ On  a  scale  0  to  100,  where  0  indicates  no  regret  and  100  indicates  the 

maximum  regret  you  could  feet,  how  would  you  rate  the  level  of  your 
regret  if  you  were  referred  to  hospice  instead  of  continue  receiving 
necessary  life-sustaining  treatment? 

Nq  Maximum 

regret  ^  regret 

_ >.100 


Thi^VfiSLie  represents  the  consequences  unnecessary  hospice  referral 


Figure  2  DVAS  (Dual  Visual  Analogue  Scales).  The  DVAS  are 
used  for  the  elicitation  of  the  decision  maker’s  threshold  probability. 


you  were  not  referred  to  hospice  but  instead  you  con¬ 
tinued  receiving  unnecessary  treatment?  That  is,  how 
much  would  you  regret  if  you  did  not  reap  the  bene¬ 
fits  of  hospice  care?  *Note  that  this  value  corre¬ 
sponds  to  Ul  -  U3. 

2.  On  the  scale  0  to  100,  where  0  indicates  no  regret 
and  100  indicates  the  maximum  regret  you  could 
feel,  how  would  you  weigh  the  level  of  your  regret,  if 
you  were  referred  to  hospice  instead  of  continue 
receiving  necessary  life-sustaining  treatment?  That  is, 
how  much  would  you  regret  if  you  sustained  harms 
from  hospice  care?  *Note  that  this  value  corresponds 
to  U4  -  U2. 

For  example,  suppose  that  the  patient  -  who  is  aware 
of  his/her  terminal  condition-  answers  50  and  25  to  the 
questions  1  and  2  respectively.  This  means  that  the 
patient  considers  50/25  =  2  times  worse  not  to  be 
referred  to  hospice  when  necessary  than  receiving  an 
unnecessary  hospice  referral.  The  threshold  probability 
for  this  patient  is  (equation  2) 

Pf  ~  i  ,  U\  -u3  I  50  ^-33  or  ^0//°  • 

1  U4—U2  25 


Regret  DCA  and  extensions 

The  clinical  problem  we  face  in  the  situation  of  hospice 
referral  is  how  to  use  reasonably  accurate  predictions  of 
death,  p,  coupled  with  the  patient’s  preferences  (as 
expressed  in  terms  of  threshold  probability,  Pt)  to  arrive 
at  the  optimal  decision  for  a  specific  individual.  The 
problem  is  decomposed  into  three  strategies:  (1)  act 
based  on  the  prediction  model  (SUPPORT)  (e.g.  refer  to 
hospice  if  p  ^  Pt  and  continue  treating  otherwise),  (2) 
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(3) 


refer  all  patients  to  hospice,  and  (3)  continue  current 
treatment  for  all  patients  (i.e.  refer  no  patients  to 
hospice). 

Each  of  these  strategies  may  inflict  physiological  and/ 
or  psychological  damages  to  the  patient.  Specifically,  a 
patient  may  suffer  harms  due  to  a  treatment  strategy  (e. 
g.  adverse  effects)  or  harms  due  to  the  prognostication 
test  (e.g.  a  test  requiring  invasive  procedure).  We 
express  these  harms  as  loss  in  utility  associated  with 
actions  we  may  undertake.  To  that  end,  we  define  HRx, 
Hhosp  and  Hte  as  the  utility  losses  due  to  harms  of  the 
treatment,  hospice,  and  prognostic  test,  respectively. 

Figure  3  presents  the  decision  tree  describing  the 
overall  hospice  referral  problem,  p  =  P(D  +)  is  the  prob¬ 
ability  that  the  patient’s  LE  is  less  than  or  equal  to  6 
months  as  estimated  by  the  prediction  model  (SUP¬ 
PORT);!  -  p  =  P(D  -)  is  the  probability  that  the  patient’s 
LE  is  greater  than  6  months,  and  U j,  i  I  [1,4],  are  the 
utilities  corresponding  to  each  of  the  decision  model 
outcomes  (detailed  in  the  previous  section).  The  vari¬ 
ables  Hosp  and  Rx  correspond  to  referring  a  patient  to 
hospice  and  continuing  current  curative  treatment, 
respectively.  Rg,  is  the  regret  associated  with  an  action, 
e.g.  Rg(Hosp,  D-)  is  the  regret  one  may  feel  if  the  patient 
was  referred  to  hospice  when  his/her  LE  was  greater 
than  6  months.  Finally,  te  designates  that  the  patient 
received  a  prognostication  test. 

Considering  the  decision  tree  in  Figure  3  we  can  com¬ 
pute  the  expected  regret  associated  with  each  decision 
in  terms  of  the  utilities  of  each  possible  outcome  as  fol¬ 
lows  (detailed  derivation  is  presented  in  the  Appendix): 


JiiWw 


fisWa.  UrHb 

Figure  3  Decision  tree  describing  the  overall  hospice  referral 
process.  In  this  figure  p  =  P  (D+):  probability  the  patient’s  LE  is 
less  than  or  equal  to  6  months;  1  -  p  =  P  (D  -):  probability  the 
patient’s  LE  is  greater  than  6  months;  Lf  i  T  [1,4]:  the  utilities 
corresponding  to  each  of  the  decision  model  outcomes;  Hosp: 
hospice  referral;  Rx  treatment  continuation;  Rg:  regret  associated 
with  an  action;  Hr*:  utility  losses  due  to  harms  of  treatment;  HHosP: 
utility  losses  due  to  harms  of  hospice;  Hte:  utility  losses  due  to 
harms  of  the  prognostic  test  (SUPPORT). 
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In  addition  to  harms,  equations  3,  4  and  5  incorporate 
the  effects  of  treatment  and  hospice  care  using  measures 
of  Relative  Risk  Reduction:  and  RRRrx  RRRhosp  respec¬ 
tively.  The  values  for  these  measures  are  treatment  spe¬ 
cific  and  can  be  acquired  from  the  literature.  We  have 
incorporated  hospice  effects  because  a  recent  study  [11] 
has  shown  that  early  palliative  care  for  patients  with 
metastatic  non-small  cell  lung  cancer  could  increase 
survival.  The  variables  TP,  FP,FN,  TN  are  related  to  the 
prognostic  capability  of  the  SUPPORT  model  (see 
appendix  for  detailed  derivation)  [24]. 

Since  the  regret  of  omission  and  regret  of  commission 
have  been  generalized  to  include  effects  and  harms 
related  to  management  strategies  and  testing,  the  func¬ 
tion  of  threshold  probability  (equation  2)  becomes: 

1 

P‘  ~  !  U,-U3+Hgx-HHv  (6) 

Where  U,  -  U3  +  Hrx  -Hhosp  corresponds  to  the  regret 
associated  with  not  referring  the  patient  to  hospice 
when  necessary,  and  U4  -  U2  -  HRx  -  Hhosp  corresponds 
to  the  regret  associated  with  unnecessary  hospice 
referral. 

Choosing  the  optimal  strategy 

The  optimal  strategy  is  selected  as  the  one  which  will 
bring  the  least  amount  of  regret.  The  regret  DCA  algo¬ 
rithm  expresses  the  regret  associated  with  each  strategy 
in  terms  of  threshold  probability  and  is  implemented  as 
follows  [24]: 

1.  Select  a  value  for  threshold  probability. 

2.  Assuming  that  patients  should  be  referred  to  hos¬ 
pice  if  p  ^  Pt  and  should  continue  current  treatment 
otherwise,  compute  #TP  and  #FP  for  the  prediction 
model. 

3.  Calculate  the  ERg(SUPPORT)using  equation  5. 
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Figure  4  Decision  curves  for  hospice  referral.  In  this  figure, 
RRRhosp  =  0,  RRRrx  =  0,  HRx=  HHosp  =  Hte  ={f.  At  threshold 
probability  equal  to  10%,  the  optimal  decision  is  refer  the  patient  to 
hospice;  at  40%  the  optimal  decisipnls  to  use  the  SUPPORT  model. 


4.  Calculate  ERg(Rx)  using  equation  4. 

5.  Compute  the  ERg(Hosp)using  equation  3. 

6.  Repeat  steps  1-6  for  a  range  of  threshold 
probabilities. 

7.  Graph  each  expected  regret  function  calculated  in 
steps  3-5  against  each  threshold  probability. 


At  each  threshold  probability,  the  action  with  the  low¬ 
est  value  of  expected  regret  corresponds  to  the  most 
desired  action.  For  example,  in  Figure  4,  at  a  threshold 
probability  equal  to  10%  (e.g.  the  patient  considers  9 
times  worse  not  to  be  referred  to  hospice  when  necessary 
than  to  receive  an  unnecessary  hospice  referral),  the  opti¬ 
mal  strategy  is  to  refer  the  patient  to  hospice. 

Figures  5,  6  and  7  depict  the  regret  associated  with 
alternative  decision  strategies  as  they  relate  to  different 


| - Treatment  all - Hospice  all  - SUPPORT  ’ 


Figure  5  Decision  curves  as  a  function  of  RRRrx.  In  this  figure, 
RRRhosp  =  0,  RRRrx  =  2  to  8%,  =  HHosp  =  Hte  =  0.  As  the  effect 

of  treatment  increases,  the  regret  associated  with  treating  all 
patients  and  with  the  SUPPORT  model  slightly  decreases.  The 
strategy  of  using  the  SUPPORT  model  to  refer  a  patient  to  hospice 
is  the  action  with  the  least  amount  of  regret  for  the  wider  range  of 
threshold  probabilities. 


RRR'Hosp=2% 


RRR-Hosp=4% 


Figure  6  Decision  curves  as  a  function  of  RRRHosP'  In  this  figure 


rrrh 


.  =  2%  to  8%,  RRRrx  =  0, 


of  hospice  care  increases,  the  regret  associated  with  hospice  and 
with  the  SUPPORT  model  slightly  decreases.  As  previously,  the 
strategy  of  using  the  SUPPORT  model  to  refer  a  patient  to  hospice 
is  the  action  with  the  least  amount  of  regret  for  a  wide  range  of 
threshold  probabilities. 


values  of  hospice  effectiveness  (Figure  5),  treatment 
effectiveness  (Figure  6),  and  harms  due  to  the  prognosti¬ 
cation  test  (Figure  7).  As  expected,  when  the  harms  due 
to  the  prognostication  test  are  increased,  then  the  area 
of  threshold  probability  at  which  the  prognostication 
model  is  the  optimal  decision  is  reduced  (Figure  5). 
Even  though,  it  is  not  expected  that  the  SUPPORT 
model  will  actually  create  harms,  at  least  physiological, 
to  the  patient,  this  is  not  always  the  case  for  other  diag¬ 
nostic  tests  that  may  be  more  invasive  (e.g.  screening 
for  prostate  cancer). 

As  can  be  seen  from  Figures  5,  6  and  7  the  optimal 
decision  is  derived  by  the  SUPPORT  model  for  a  rather 
wide  range  of  threshold  probabilities.  Therefore,  it 
appears  that  the  SUPPORT  model  is  the  superior 


- Treat  all  —  ■  SUPPORT  Harms  10% 

Figure  7  Decision  curves  as  a  function  of  Hte.  In  this  figure, 
RRRhosp  =  0,  RRRrx  =  0,  FIRx=  HH0sp  =  0  and  Flte  =  0%  and  10%.  As 
the  harms  associated  to  the  prediction  test  increase,  so  does  the 
expected  regret  of  utilizing  the  SUPPORT  model  for  hospice  referral. 
Increasing  the  harms  due  to  treatment  or  due  to  hospice  care  does 
not  have  an  effect  on  the  decision  curves. 
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strategy  for  the  vast  majority  of  decision  makers,  regard¬ 
less  the  effects  of  the  alternative  management  strategies. 
However,  since  the  threshold  probability  expresses  the 
personal  preferences  of  a  particular  decision  maker,  it  is 
not  unusual  for  specific  patients  to  have  smaller  or 
greater  threshold  probability  values  than  the  majority  of 
decision  makers.  This  is  the  power  of  the  proposed 
methodology,  which  allows  for  decision  making  at  the 
individual  level.  For  example,  if  the  decision  maker  pre¬ 
sents  a  threshold  probability  greater  than  *92%,  the 
optimal  decision  would  be  to  continue  life-sustaining 
treatment  even  if  it  is  deemed  not  to  be  effective(Figure 
4).  Similarly,  for  small  values  of  threshold  probability, 
the  desired  action  would  be  to  refer  the  patient  to 
hospice. 

Decision  Support  System 

As  our  theoretical  discussion  highlighted,  decisions 
about  life  and  death  are  complex  and  difficult  at  both 
the  emotional  and  cognitive  level.  Therefore,  it  is  not 
surprising  that  the  SUPPORT  model  originally  failed  to 
improve  the  quality  of  care  for  terminally  ill  patients 
despite  its  reasonable  accuracy  in  prediction  of  probabil¬ 
ity  of  survival  [17,20].  Any  attempt  to  focus  on  a  single 
dimension  of  the  complex  hospice  referral  process  is 
not  likely  to  succeed.  An  accurate  prognostic  model  is 
only  the  first  step.  Having  the  apparatus  to  take  into 
account  trade-offs  associated  with  the  hospice  referral 
decision  while  taking  into  consideration  the  patients’ 
preferences  represent  further  necessary  steps  to  improve 
the  care  of  terminally  ill  patients.  In  addition,  we 
hypothesize  that  the  SUPPORT  intervention  failed 
because  it  was  not  available  at  the  point  of  care  in  real 
time.  This  is  because  the  most  desired  outcomes  are 
best  achieved  when  decision-making  occurs  in  real-time, 
at  the  point  of  care  [33,34]. 


To  facilitate  the  decision  making  process  for  the  hos¬ 
pice  referral  at  bedside,  we  propose  a  web-based  clinical 
decision  support  system  (CDSS)  that  computes  the 
probabilities  of  survival  and  death  for  individual  patients 
using  the  SUPPORT  model,  elicits  personal  preferences 
from  patients  and/or  physicians,  and  utilizes  regret  DCA 
to  suggest  the  optimal  decision  for  a  particular  patient. 

Features 

Access 

Our  goal  is  to  develop  a  CDSS  that  can  be  accessed  by 
everyone  and  from  anywhere  regardless  the  operating 
system  one  uses.  At  the  same  time,  it  is  desirable  to 
develop  a  system  that  can  eventually  be  integrated  with 
various  healthcare  providers’  electronic  medical  records 
(EMR).  We  concluded  that  a  web-based  implementation 
would  fulfil  such  requirements. 

Data  storage 

The  CDSS  performs  the  required  computations  without 
retaining  or  transmitting  sensitive  and  identifiable 
information. 

Results 

In  this  section  we  present  a  prototype  of  the  CDSS, 
developed  to  demonstrate  the  applicability  of  our  theo¬ 
retical  framework  for  hospice  referral.  Each  subsection 
describes  the  results  of  the  methods  shown  in  the  pre¬ 
vious  section  in  conjunction  with  the  description  of  the 
corresponding  module.  Figure  8  depicts  the  logical  dia¬ 
gram  that  outlines  the  operation  of  the  CDSS.  Briefly, 
the  operation  begins  by  interviewing  the  patient  or  sur¬ 
rogate  to  elicit  his/her  threshold  probability.  Based  on 
the  value  of  threshold  probability  equation,  the  optimal 
strategy  for  the  particular  patient  is  derived  (e.g.  refer  to 
hospice,  continue  treatment,  or  use  of  prediction 
model).  If  the  optimal  strategy  is  to  follow  the 


Annual  progress  report  page  6 


prediction  model  (SUPPORT),  then  using  the  equations 
Al,  A2  and  1,  the  probabilities  of  survival  and  death  are 
computed  for  the  particular  patient.  The  probability  of 
death  is  then  contrasted  with  the  patient’s  threshold 
probability  and  the  optimal  decision  is  derived  (refer  to 
hospice,  or  continue  treatment).  At  each  step  described, 
the  patient  selects  the  level  of  information  he/she  wishes 
to  be  exposed  to.  For  example,  the  patient  may  not  wish 
to  know  his/her  threshold  probability  or  probability  of 
death.  Instead  he/she  wishes  to  know  only  the  optimal 
decision  regarding  his/her  condition. 

General  implementation  details 

The  proposed  CDSS  is  a  web-based  application  residing 
on  the  USF  Health  servers.  The  web  address  is  http:// 
health.usf.edu/research/ebm/decisionaids.htm.  It  has 
been  developed  based  on  the  Adobe®  ColdFusion® 
application  technology  and  the  interface  has  been 
designed  using  html  and  JavaScript  programming  lan¬ 
guages.  The  hardware  and  software  requirements  from 
the  user’s  point  of  view  are  modest.  The  system  mns  on 
any  contemporary  computer  with  net  browsing  capabil¬ 
ities.  However,  at  this  stage  the  CDSS  is  not  optimized 
for  use  with  handheld  devices.  The  CDSS  consists  of  3 
different  modules  as  described  below. 


Elicitation  of  threshold  probability  module 
The  threshold  elicitation  module  consists  of  the  dual 
visual  analogue  scales,  used  to  weigh  the  patient’s  regret 
in  the  case  of  wrong  decisions.  Each  scale  has  100 
points  where  0  corresponds  to  no  regret  and  100  to 
maximum  regret.  Depending  on  the  role  of  the  decision 
maker  (e.g.  patient/surrogate  or  physician)  two  different 
sets  of  questions  are  displayed.  These  questions  are 
designed  to  capture  the  regret  of  omission  and  the 
regret  of  commission.  For  the  remainder  of  this  paper, 
we  assume  that  the  decision  maker  is  the  patient.  As  in 
pain  scales  [35],  each  visual  analogue  scale  uses  facial 
expressions  to  graphically  represent  variations  in  regret 
(Figure  9).  A  summary  of  the  decision  maker’s  prefer¬ 
ences  is  presented  for  final  verification.  The  threshold 
probability  for  the  particular  patient  is  derived  using 
equation  2,  however  is  not  displayed  until  the  decision 
maker  requests  it. 

Decision  module 

The  decision  module  utilizes  the  decision  maker’s 
threshold  probability  and  the  regret  DCA  methodology 
to  derive  the  optimal  decision.  For  example,  the  prefer¬ 
ences  of  the  patient  depicted  in  Figure  9,  correspond  to 
a  threshold  probability  equal  to  29%.  From  Figure  4  the 
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Figure  9  Elicitation  of  threshold  probability,  The  user  (patient/surrogate/physician)  weighs  the  two  alternative  management  strategies  in 
terms  of  regret. 
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strategy  that  will  bring  the  least  amount  of  regret  is  to 
use  the  prognostication  model  (SUPPORT)  for  the  hos¬ 
pice  referral  recommendation.  In  this  case,  the  decision 
module  initiates  the  SUPPORT  module. 

SUPPORT  module 

If  the  optimal  strategy  derived  by  the  decision  module  is 
to  utilize  the  prognostication  model,  the  SUPPORT 
module  is  enabled.  This  module  (Figure  10)  is  used  to 
compute  the  probability  of  death  for  the  particular 
patient  based  on  the  SUPPORT  prognostication  model. 
Currently,  the  user  inserts  all  required  information  to 
the  CDSS.  In  the  future,  this  information  will  be  cap¬ 
tured  automatically  from  the  health  care  provider’s  elec¬ 
tronic  medical  records  system.  Data  validation 
restrictions  have  been  imposed  to  protect  the  integrity 
of  the  collected  data. 

Once  the  values  of  all  available  variables  have  been 
inserted  in  the  corresponding  cells,  the  patient’s  life 
expectancy  and  probabilities  of  survival  and  death  are 
computed.  The  decision  module  is  employed  again  to 
display  the  optimal  recommendation. 

Decision  justification  module 

The  decision  justification  module  explains  in  detail  and 
at  the  user’s  request  the  reasons  that  led  to  a  particular 


recommendation  (Figure  11).  It  contains  information 
regarding  the  decision  maker’s  threshold  probability,  the 
optimal  strategy  associated  with  the  threshold  probabil¬ 
ity  and  the  patient’s  probability  of  death  (if  applicable). 
Since  people  often  misinterpret  probabilities  [36],  we 
complement  the  results  presented  in  terms  of  probabil¬ 
ities  using  frequency  format  (Figure  11).  The  latter  for¬ 
mat  is  currently  considered  the  best  way  to  represent 
favorable  and  unfavorable  facts  regarding  medical  inter¬ 
ventions  [37].  The  justification  module  is  highly  techni¬ 
cal  and  should  only  be  reviewed  by  decision  makers 
who  wish  to  know  more  about  their  or  their  patient’s 
condition. 

Case  Study 

Figure  12  summarizes  the  decision  process  for  a  patient 
whose  information  is  simulated  in  Figures  10,  11  and 
12.  The  probability  of  death  and  the  threshold  probabil¬ 
ity  of  this  patient  have  been  computed  as  85%  and  29% 
respectively.  At  a  29%  threshold  probability,  the  optimal 
strategy  is  to  use  the  prediction  model  for  hospice  refer¬ 
ral  (Figure  4).  Therefore,  since  p  >Pt  the  patient  should 
be  referred  to  hospice.  For  completeness,  all  possible 
decision  routes  are  depicted  in  Figure  12.  The  route  cor¬ 
responding  to  the  specific  simulated  patient  is  shown 
using  bold  arrows. 
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Figure  10  SUPPORT  user  interface.  The  user  enters  all  information  regarding  the  particular  patient  to  compute  the  probability  of  death  and 
survival  within  the  next  6  months.  LE  results  are  presented  to  the  patient  through  the  decision  justification  module  after  the  patient’s  request. 
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Figure  11  Justification  of  the  hospice  referral  recommendation.  The  particular  patient  depicted  has  29%  threshold  probability  at  which  the 
optimal  strategy  is  derived  by  the  SUPPORT  model.  The  patient  has  85%  probability  of  death  in  the  next  180  days.  Therefore,  the  optimal 
decision  is  to  be  referred  to  hopsice. 


Discussion 

In  this  article  we  describe  both  the  theory  and  applica¬ 
tion  behind  a  hospice  referral  clinical  decision  support 
system.  To  the  best  of  our  knowledge,  this  is  the  first 
CDSS  that  integrates  two  well  established  methodolo¬ 
gies,  one  for  prognostication  (SUPPORT)  and  the  other 
for  decision  making  (regret  DCA),  to  assist  with  the  hos¬ 
pice  referral  decision-making  process. 

The  recently  developed  regret  DCA  incorporates  the 
decision  maker’s  preferences  towards  alternative  man¬ 
agement  strategies  from  the  perspective  of  regret  theory 
in  terms  of  threshold  probability.  Such  an  approach  pro¬ 
motes  personalized  patient  care.  We  anticipate  that  the 
regret-based  approach  is  more  appropriate  for  the  hos¬ 
pice  referral  process  than  other  preference  elicitation 
techniques,  due  to  the  nature  of  the  problem  where 


there  are  really  no  optimal  options  available-  the  opti¬ 
mal  decision  can  be  only  considered  as  the  one  with  the 
least  regret. 

Modem  cognitive  theories  increasingly  focus  on  the  so 
called  dual-processing  theory  in  which  both  intuition 
(system  1)  and  analytical,  deliberative  process  (system  2) 
are  important  for  balancing  risks  and  benefits  in  the 
decision-making  process  [38].  We  believe  that  rational 
decision-making  should  take  into  account  both  formal 
principles  of  rationality  and  human  intuition  about  good 
decisions[24,39,40].  One  way  to  accomplish  this  is  to 
use  regret,  a  cognitive  emotion,  to  serve  as  the  link 
between  systems  1  and  2  [24].  By  taking  into  account 
the  consequences  of  our  actions  as  well  as  the  circum¬ 
stances  under  which  we  can  live  with  our  mistakes  we 
anticipate  that  the  goal  of  reconciling  the  formal 
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Table  1  Values  of  Survival  (S)  as  described  in  equation 
A2  for  different  disease  types  and  varying  survival  times 


t 

^ARF/MOSF 

ScOPD/CHP/Cirrhosis 

Scoma 

c 

°Cancer 

0 

0.994 

0.998 

0.993 

0.993 

30 

0.691 

0.889 

0.630 

0.578 

60 

0.601 

0.837 

0.609 

0.407 

90 

0.562 

0.800 

0.581 

0.264 

120 

0.532 

0.772 

0.569 

0.190 

1  50 

0.508 

0.751 

0.551 

0.135 

177 

0.493 

0.733 

0.545 

0.108 

principles  of  rationality  and  human  intuitions  about 
good  decisions  can  be  met  [24,29,39,40].  This  is  particu¬ 
larly  true  in  the  situation  of  terminally  ill  patients. 

Our  web-based  CDSS  reflects  modern  cognitive  the¬ 
ories  to  facilitate  integration  of  the  decision-making 
ingredients  necessary  for  hospice  referral  decisions.  The 
CDSS  encapsulates  all  required  information  for  the  hos¬ 
pice  referral  process  into  a  flexible  software  that  can  be 
used  at  bedside.  Obviously,  hospice  referral  decisions 
are  complex  and  must  be  exercised  with  full  compassion 
and  deliberation.  We  advise  against  the  use  of  our  sys¬ 
tem  as  an  automatic  decision  making  tool  that  by-passes 
important  personal  interactions  between  the  patient  and 
his/her  physician.  It  is  important  to  stress  that  the  elici¬ 
tation  of  the  threshold  probability  as  described  herein 
reflects  the  belief  (also  captured  in  recent  legislation 
[41])  that  patients  and  their  families  want  to  be  told  the 
“truth”  about  the  patients’  terminal  sickness  [22,41,42] 
and  that  physicians  have  ethical  obligations  to  share  this 
information  with  patients  and  their  families  [41,42].  Our 
system  should  be  understood  as  an  aid  to  facilitate  deci¬ 
sions  in  terminal  phases  of  patient  lives. 

Our  approach  has  limitations  as  well.  The  main  lim¬ 
itation  of  the  proposed  system  remains  the  complexity 
of  the  SUPPORT  model.  Currently,  the  system  still 
requires  manual  entry  of  data.  In  addition,  failure  to 
enter  all  data  can  jeopardize  the  accuracy  of  prediction 
and  therefore,  the  decision  process.  To  cope  with  this 
limitation,  we  plan  to  integrate  our  system  into  various 
health  providers’  EMRs.  Based  on  each  EMR,  specifically 
designed  queries  will  be  used  to  retrieve  lab  values  and 
patient  demographics  to  be  fed  automatically  into  our 
system;  a  process  that  will  reduce  the  amount  of  missing 
values  and  input  errors. 

The  second  limitation  of  the  proposed  system  is  that 
empirical  data  are  not  available  to  assess  how  the  system 
actually  works  in  practice.  While  we  plan  to  undertake 
empirical  testing  of  the  system  described  here,  we 
believe  that  a  strong  theoretical  underpinning  will 
enable  better  hospice  referral  decisions  even  in  the  cur¬ 
rent  form.  This  is  because  our  system  will  essentially 


operationalize  the  decision-making  process,  which  is 
supposed  to  occur  in  every  day  practice.  Nevertheless, 
we  need  to  firstly,  identify  the  system’s  feasibility  in  real 
life  settings  and  ultimately,  if  it  appears  to  be  usable  and 
assessed  favourably  by  all  those  involved  in  the  hospice- 
referral  decision-making  process,  to  test  it  in  rando¬ 
mized  controlled  trials  against  traditional  care. 

Our  future  plans  include  both  empirical  testing  and 
implementation  of  multiple  additional  prognostication 
models  which  will  be  used  in  parallel  to  assess  optimal 
decisions  regarding  hospice  referral  and  take  advantage 
of  the  regret  DC  A  methodology.  We  anticipate  that  for 
a  different  range  of  threshold  probabilities  these  models 
may  perform  better  than  the  SUPPORT  model.  Further¬ 
more,  our  intent  is  to  develop  a  separate  version  of  our 
CDSS  optimized  for  mobile  devices. 

Conclusions 

In  this  work  we  have  presented  the  theoretical  frame¬ 
work,  accompanied  by  the  associated  CDSS,  to  facilitate 
end  of  life  care  decisions.  Our  work  combines  the  prog¬ 
nostication  power  of  the  SUPPORT  model,  the  simpli¬ 
city  of  the  DVAS  methodology  in  eliciting  people’s 
preferences  and  the  effectiveness  of  regret  DCA  at  eval¬ 
uating  alternative  management  strategies  to  resolve  the 
dilemma  of  choosing  traditional  vs.  palliative  care  for 
patients  at  terminal  stages.  A  clinical  evaluation  of  the 
CDSS  is  planned. 

Appendix 

Support  implementation 

SUPPORT  is  implemented  in  two  steps.  First,  the  SUP¬ 
PORT  physiology  score  is  computed  based  on  equation 
A1  [22]. 

SPS  =  259.9 {ARF/MOSF}  +  263  A{COPD/CHF] 

+  241  A{Cirrhosis/  Coma } 

+  281 .5  {Lung/ ColonCancer} 

-  0.06174mm(Pa02/Fi02,  225) 

-  0.631 6 min (MeanBP,  60) 

+  1.0205 WBC  -  0.367 6(WBC  -  8)+ 

-  0.5631  {WBC  -  11)+  +  0.2691  min(Alb,  4.6) 

+  0.2312  Aresp  —  2.362Temp  +  1.32 6(Temp  —  36.6)+ 

+  2.473(Temp  -  38.3)^-  1.579  x  10“  X HR 
+  9.770  x  10-5  (HR  —  55)+  —  2.189  x  10-4  (HR  -  80)^ 

+  1.518  x  10 ~4(HR  -  110)3+ 

5  ,  (Al) 

-  3.062  x  10“  (HR  -  149 )3  +  0.97635*7  v  7 

-  0.7481(5*7  -  7)+  -  6.87610 

+  11.6058(0  -  0.600)+  -  21.8413(0  -  1.000)+ 

+  10.3574(0  -  1.500)3  -  0.1219(0  -  5.399)1 

-  0.6167096 No  +  0.0021118(TVa  -  128)3 
-0.0036730 (Na  -  135)+  +  0.00061 26(Na  -  139)+ 

+  0.0009486(7Va  —  148)3 

-  6.278{  COPD/CHF }  X  min(Alb,4.6) 

-  11 .45{  Lung/ ColonCancer)  x  min(^4/Z*,  4.6) 

+  [ARF/MOSF]  [-2.3549  WBC 

+  2.7494 (WBC  -  8)+  -  0.4638 (WBC  -  11)+] 
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where: Alb:  albumin;  Aresp:  APACHE  III  respiration 
score;  Bil:  bilirubin;  Cr:  Creatinine;  Na:  sodium;  Pa02: 
partial  pressure  oxygen  in  arterial  blood;  MeanBP:  mean 
arterial  blood  pressure;  WBC:  white  blood  cell  count  in 
thousands;  Temp:  temperature  in  Celsius;  HR:  heart  rate 
per  minute;  ARF:  Acute  respiratory  failure;  MOSF:  Mul¬ 
tiple  organ  failure;  Cirrhosis:  Cirrhosis;  Coma:  Coma; 
Lung:  Lung  cancer;  ColonCancer:  Colon  cancer;  COPD: 
Chronic  obstmctive  pulmonary  disease;  CHF:  Congestive 
heart  failure.  Also: 

r 

1 ,  if  patient  in  the  disease  group 

{< disease  group}  = 

0,  otherwise 

r 

x,  if  x  >  0 

(x)+  = 

0,  otherwise 

WBC  =  9,  if  WBC  <  9  and  {disease  group  }  ^  ARF 
/MOSF 

WBC  =  40,  if  WBC  >  40 

Cr  =  15,  if  Cr  >15 

The  second  step  in  implementing  the  SUPPORT 
model  is  to  calculate  the  probability  of  survival  for  the 
individual  patient  based  on  equation  A2  [22]. 

P{T  >  £ | disease  group  =  i]  =  Sf(t)e  *  (A2) 

where  T:  survival  time  in  days;  t:  arbitrary  time;  S 
described  in  Table  1  [22]  and 

X'b  =  -3.652  +  0.8356{  CNF}  +  0.9257{  Cirrhosis} 

+  0.6287 {LungCancer}  ±  1. 18 03 {  MOSFw/Malig} 

+  0.01434* Scoma  ±  0.01935Age  +  0.2413 Cancer 

-  1.863  Hday  +  3.4  "  1  +  0.081 21 SPS 

+Age[ 0.015261{  COPD  / CHF / Cirrhosis} 

+  0.009047{  Coma}  —  0.008294{  Cancer}  ] 

+Age[- 0.012498{  CHF}  -  0.004578{  Cirrhosis} 

-  0.001435{  LungCancer} 

-  0 . 0 1 3  8  91  {  MOSFw  / Malig } 

where  Scoma:  SUPPORT  coma  score  (0-100); 
MOSFw/Malig:  Multiple  organ  failure  with  malignancies; 
Hday:  day  in  hospital  when  qualified  for  study;  Cancer: 
Cancer  by  comorbidity  or  primary  disease  category  (0  = 
no;  1  =  present;  2  =  metastatic)  [22]. 

Derivation  of  the  Expected  Regret  functions 
As  outlined  in  the  Introduction,  seriously  and  terminally 
ill  patients  may  reap  a  number  of  benefits  by  the  hos¬ 
pice  program.  Nevertheless,  after  enrollment  into  hos¬ 
pice,  the  patient  (or  the  family,  or  the  physician)  may 


feel  that  this  was  a  wrong  decision,  and  subsequently 
may  regret  it.  Similarly,  the  patient  may  feel  regret  for 
the  treatment  that  he/she  continues  to  receive  because 
it  is  unnecessary,  inappropriate,  and/or  harmful.  Figure 
3  represents  our  hospice  decision  tree  in  terms  of  regret 
from  which  we  can  compute  the  expected  values  of 
regret  associated  with  each  strategy  as  follows: 

ERg[Hosp]  = 

(A3) 

(1  -p)*(U4  ~  U2  -  H^  -  HHosp) 

ERg[Rx]  =p*(Ui  -  U3+  HRx  -  HHosp)  (A4) 

ERg[SUPPORT]  =p*TP*H,e 
+  (1  -p)  *  FP 

*(U4  -  lh  ~  (Hrx  -  Hhosp )  +  Hte)  (A5) 

+p*FN  *  (Ui  -U3+  (. Hu- 1  ~  HHosp)  +  Hte) 

+(1  —  p)  *  TN  *  Hte 

The  variables  TP,  FP,TN,  FN  are  related  to  the  prob¬ 
abilities  p  (p  >  pt  n  d  +),  p  (p  >  pt  n  d  -),  p  (p  <pt  n  d 

-)  and  P  (p  <Pt  f!  D  +)  respectively,  and  are  estimated  as 
follows: 

•  P  (p  >  Pt  fl  D  +)  *  the  number  of  patients  who  will 
die  within  6  months  and  for  whom  the  prognostic 
probability  is  greater  than  or  equal  to  Pt  (with  #TP 
=  number  of  patients  with  true  positive  results, 
P(p  >  Pt  n  D  +)  ~  ® ,  where  n  is  the  total  number 
of  patients  in  the  study). 

•  P  (p  >  Pt  fl  D  -)  ~the  number  of  patients  who  will 
survive  for  longer  than  6  months  and  for  whom  the 
prognostic  probability  is  greater  than  or  equal  to  Pt 
(with  #FP  =  number  of  patients  with  false  positive 

results,  P(p  >  Pt  n  D— )  ~  ^) . 

•  P  (p  <Pt  fl  D  +)  ~the  number  of  patients  who  will 
die  within  6  months  and  for  whom  the  prognostic 
probability  is  less  than  Pt  (with  #FN  =  number  of 
patients  with  false  negative  results, 
P(p  <P,n  D+)  =  “). 

•  P  (p  <Pt  fl  D  -)  «the  number  of  patients  who  will 
survive  for  longer  than  6  months  and  for  whom  the 
prognostic  probability  is  less  than  Pt  (with  #TN  = 
number  of  patients  with  true  negative  results, 
P{p  <PtnD- )  - 

To  incorporate  the  effects  of  alternative  treatments  (e. 
g.  treatment  and  hospice  care)  in  equations  A3-A5  we 
use  the  Relative  Risk  Reduction  reported  in  literature  for 
each  strategy  as  follows: 
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ERg[Hosp]  =  (1  -p)  *  (1  -  RRRhosp) 
*(7/4  -U2  -Hr,  -HHosp) 

ERg[Rx]=p*(  1  -RRR&) 

HUi  -u^+Hr,  -hHosp) 


(A6) 


(A7) 


ERg[SUPPORT]  = 
p  *  (1  “  RRRhosr)  *  TP  *  Hte 
+(1  — /?)  *  (1  -  RRRhosp)  *  FP 

*(7/4  -U2-  (Hr,  -  HHosp)  +  Hte)  (A8) 

+p  *  (1  —  //TR^)  *  7W 

*(T/l  “  ^3  +  (Hrx  ~  Huospe)  +  Hte) 

+(1  -p)  *  (1  -  RRRrx)  *  77V  *  Hte 

Since  TP  +  FN  =  1  and  FP+  TN  =  1,  we  have: 
p  *  TP+  (1  —  /?)  *  FP  +/?  *  /TV 
+(1  -/?)  *  TN  =/>  +  (l  -/?)=  1 

Therefore,  equation  A8  becomes: 

/Wg[V7///OW/]  = 

(1  -  p  *  RRRHosp  *  TP  -  (1  -  /?)  *  RRRhosp  *  T^P 
-p  *  RRRrx  *  /TV  -  (1  —  /?)  *  RRRr,  *  77V)  *  Hte 
+n  -p)  *  (i  -  *  /sp  (A9) 

*  7/4  -7/2  -  (/fee  -HHosp) 

*  (1  -  RRRrx)  *  /TV 

*  7/i  —  7/3  +  (ife*  —Hnosp) 

Scaling  the  equations  A3,  A4  and  A9  with  the  quantity 
(l^  -  U3  +  Hrx  -  HHosp)  and  replacing  the  expression 

Utt  %  |r^  with  tAt,  we  derive  the  final  equations 

u\  -U3+MRx-MHosp  1  l  t 

for  the  expected  regret  (equations  3,  4,  and  5). 
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Patient  Data  Collection  Form 


Patient  ID# 

Physician  ID# 

Demographics 

Who  is  being  interviewed?  (circle  one)  Patient  Surrogate 


Date  of  Birth: 

Date  of  Death: 

Date  of  TGH  Admission: 

Gender  (check  one) 

l:  g]  Male 

jg]  Female 

Ethnicity  (circle  one):  White 

African  American 

Hispanic 

Asian  Pacific  Islander 

Other: 

Diagnosis 

*  Primary  Diagnosis: 

Date  of  Diagnosis: 


Patient  Directives  (circle  one): 


Do  Not  Resuscitate  (DNR) 


^  Days  in  hospital  when  qualified  for  study  (please  circle  one);  if  more  than  1 0,  write  out. 

123456789  10  Other: 


*  Number  of  days  in  hospital: 


Full  Code 


* 

* 

* 

* 

* 


Vital  Signs 

Value 

Date  Collected 

Systolic  BP 

mmHg 

Diastolic  BP 

mmHg 

Body  Temperature 

F 

Heart  Rate 

BPM 

Respiratory  Rate 

RR 

■* 

■* 

■* 

■* 


■* 


■* 


* 


Laboratory  Data 

Value 

Date  Collected 

Serum  Creatinine 

mg/dl 

Serum  Bilirubin 

mg/dl 

Serum  Sodium 

mEq/lt 

Serum  Albumin 

g/dl 

Blood  Glucose 

mg/dl 

Hematocrit 

% 

Blood  pH 

24  Hour  Urine  Output 

Cc/day 

White  Blood  Cell  Count 

K/UL 

Alveolo-arterial  p02 
gradient  (A-aD02) 

Blood  Urea  Nitrogen 
(BUN) 

mg/dl 

Partial  Pressure  Carbon  Dioxide 
in  arterial  blood  (PaC02) 

mmHg 

Partial  Pressure  Oxygen 
in  Arterial  Blood  (Pa02) 

mmHg 

Fraction  of  Inspired 
Oxygen  in  a  Gas  (FI02) 
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Coronary  Artery  Bypass  Graft  (CABG)  information  (if  applicable) 


Emergency  Surgery  Yes  No 


Ejection  Fraction 


Diabetes  (medication  dependent) 

Yes 

No 

CABG  was  redone 

Yes 

No 

Number  of  grafts 

Internal  mammary  artery  used 

Yes 

No 

Isosorbide  mononitrate  used 

Yes 

No 

Co-morbidities 


Acute  renal  failure/insufficiency 

Yes 

No 

Acquired  immunodeficiency  syndrome 

Yes 

No 

Cancer 

Yes 

No 

Chronic  obstructive  pulmonary  disease 

Yes 

No 

Congestive  heart  failure 

Yes 

No 

Cirrhosis 

Yes 

No 

Colon  cancer 

Yes 

No 

Coma 

Yes 

No 

Hepatic  failure 

Yes 

No 

lmmunosuppression(HIV,Trpl,  neutrophil  <1  k/uL) 

Yes 

Lung  cancer 

Yes 

No 

Lymphoma 

Yes 

No 

Leukemia 

Yes 

No 

Multiple  myeloma 

Yes 

No 

Multiple  organ  system  failure  w/  cancer 

Yes 

No 

Multiple  organ  system  failure  w/  sepsis 

Yes 

No 

Metastasis 


No 


^Calculate  using  tables  on  page  3. 


Glasgow  Coma  Score: 


*  PPS: 


Other  co-morbidities 


Patient  interested  in  knowing  the  life  expectancy  calculated  using  the  prognostication  models?  Yes  No 

Life  expectancy  estimate  shared  with  patient?  Yes  No 


Patient  Data  Collection 
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Glasgow  Coma  Scale 

The  Glasgow  Coma  Scale  provides  a  score  in  the  range  3-1 5;  patients  with  scores  of  3-8  are  usually  said  to  be  in  a  coma.  The  total  score  is 
the  sum  of  the  scores  in  three  categories.  For  adults  the  scores  are  as  follows  (please  circle  on  in  each  category): 


Eye  Opening 
Response 

Spontaneous:  open  with  blinking  at  baseline 

4  points 

Opens  to  verbal  command,  speech,  or  shout 

3  points 

Opens  to  pain,  not  applied  to  face 

2  points 

None 

1  point 

Verbal  Response 

Oriented 

5  points 

Confused  conversation,  but  able  to  answer  questions 

4  points 

Inappropriate  responses,  words  discernible 

3  points 

Incomprehensible  speech 

2  points 

None 

1  point 

Motor  Response 

Obeys  commands  for  movement 

6  points 

Purposeful  movement  to  painful  stimulus 

5  points 

Withdraws  from  pain 

4  points 

Abnormal  (Spastic)  flexion,  decorticate  posture 

3  points 

Extensor  (rigid)  response,  decerebrate  posture 

2  points 

None 

1  point 

Palliatative  Performance  Scale  (Please  circle  one  PPS  Level) 


PPS  Level 

Ambulaton 

Activity  &  Evidence  of  Disease 

Self-care 

Intake 

Conscious  Level 

100% 

Full 

Normal  activity  &  work,  no 
evidence  of  disease 

Full 

Normal 

Full 

90% 

Full 

Normal  activity  &  work,  some 
evidence  of  disease 

Full 

Normal 

Full 

80% 

Full 

Normal  activity  with  effort, 
some  evidence  of  disease 

Full 

Normal  or  reduced 

Full 

70% 

Reduced 

Unable  normal  job  or  work, 
significant  disease 

Full 

Normal  or  reduced 

Full 

60% 

Reduced 

Unable  hobby/housework, 
significant  disease 

Occasional 
assistance  necessary 

Normal  or  reduced 

Full  or  confusion 

50% 

Mainly  sit/lie 

Unable  to  do  any  work, 
extensive  disease 

Considerable 
assistance  necessary 

Normal  or  reduced 

Full  or  confusion 

40% 

Mainly  in  bed 

Unable  to  do  most  activity, 
extensive  disease 

Mainly  assistance 

Normal  or  reduced 

Full  or  drowsy  +/- 
confusion 

30% 

Totally  bed  bound 

Unable  to  do  any  activity, 
extensive  disease 

Total  care 

Normal  or  reduced 

Full  or  drowsy  +/- 
confusion 

20% 

Totally  bed  bound 

Unable  to  do  any  activity, 
extensive  disease 

Total  care 

Minimal  sips 

Full  or  drowsy  +/- 
confusion 

10% 

Totally  bed  bound 

Unable  to  do  any  activity, 
extensive  disease 

Total  care 

Mouth  care  only 

Drowsy  or  coma  +/- 
confusion 

0% 

Death 

- 

- 

- 

- 

Patient  Data  Collection 
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Physician  Demographics 


Physician  ID  # 

Gender  (check  one):  □  Male  □  Female 

Ethnicity  (circle  one):  White  African  American  Hispanic  Asian  Pacific  Islander 


Medical  school  name: 


Medical  practice  specialty: 


Years  in  practice: 


Average  number  of  patients  seen  per  year: 


Average  number  of  patients  referred  to  hospice  in  last  5  years: 


Is/has  the  physician  using/used  a  prognostication  model  for  calculation  of  life  Yes  No 
expectancy  for  his/her  patients? 


Name(s)  of  prognostication  model(s) 
being  used: 


Other: 
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Physician  Interview 


Physician  ID# 


Date  of  Interview: 
Patient  ID# 


Physician's  prediction  of  survival  (Please  circle  one):  Hours-days  Days-weeks  <  6  months 


<  12  months 


Please  circle  responses 

Physician  interested  in  knowing  the  life  expectancy  calculated  using  the  prognostication  models?  Yes  No 

Life  expectancy  estimate  shared  with  physician?  Yes  No 


Usefulness  of  prognostication  calculations  (please  circle  responses) 

Do  you  find  our  calculations  regarding  best  advice  in  terms  of  continuing  treatment/hospice  care  for  your  patient  useful? 
Yes  Somewhat  No 


How  strongly  will  these  calculations  influence  your  decision  considering  hospice  referral  for  your  patient? 

Strongly  Moderately  strongly  Somewhat  Not  at  all 

How  strongly  do  you  agree  with  our  best  advice  regarding  continuing  current  treatment  vs.  choosing  hospice  care  for  your  patient? 

Strongly  Moderately  Somewhat  Neither  agree,  Somewhat  Moderately 
agree  agree  agree  nor  disagree  disagree  disagree 

Based  on  what  you  have  talked  about  today,  would  you  choose  hospice  care  for  your  patient? 

Yes  Need  more  time  to  Need  more  time  to  Need  more  time  to  No  Other  (please  specify): 

think  it  over  discuss  with  family  discuss  with  physician 
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Appendix  9  Study  scripts 


Introductory  Script 


Good  morning/afternoon.  My  name  is _ .  Dr. _ (primary 

attending)  told  us  you  might  be  interested  in  participating  in  our  research  study  aimed  at 
helping  you  choose  what  type  of  medical  care  you  want  to  have  during  the  upcoming  months. 

Question:  Would  you  like  to  hear  more  about  our  study?  If  no.  STOP  NOW. 

Question:  What  has  your  doctor  told  you  about  your  illness?  IJjjgtierrt  does  not  understand 
that  s/he  has  terminal  disease .  STOP  NOW. 

I  understand  that  you're  going  through  a  difficult  time  right  now.  One  day  most  of  us  will  likely 
find  ourselves  in  a  situation  similar  to  the  one  you  find  yourself  in  today.  Most  people  in  this 
situation  are  uncertain  how  they  want  to  spend  the  remainder  of  their  lives.  We  have  created  a 
tool  to  help  people  and  their  doctors  make  the  choices  that  they  feel  are  right  for  them.  The 
tool  is  designed  to  help  patients  like  yourself  decide  whether  to  continue  treatment  targeting 
their  disease,  although  the  treatment  may  create  additional  discomfort,  or  whether  to  choose 
care  through  hospice  focused  more  on  providing  comfort  and  symptomatic  relief.  Either  of 
these  approaches  may  or  may  not  prolong  your  life. 

Question:  Did  your  doctor  mention  that  you  might  want  to  consider  hospice  at  this  point?  [f 
no,  STOP  NO W  and  notify  referring  physician. 

Question:  Have  you  been  contacted  by  hospice  staff?  Have  you  signed  up  for  hospice?  /f  yes, 
DO  NOT  PROCEED  UNLESS  the  patient  and,  referring  physician  have  requested,  us  to  do  so  (e.a., 
patient  has  doubt,  wants  to  clarify,  his/her  values ,  etc). 

Record  answer  on  Checklist. 


Question:  Do  you  know  what  hospice  is?  Hospice  is  a  type  of  care  that  provides  support  to 
people  at  the  end  of  their  lives.  It  can  help  people  get  their  physical,  emotional,  and  spiritual 
needs  met. 

If  you  choose  to  participate  in  our  study,  I  will  ask  you  to  sign  an  informed  consent  form.  It  will 
allow  me  to  take  information  from  your  medical  records  to  help  assess  your  current  condition 
better.  Your  name  and  other  identifying  information  will  not  be  used.  I  will  then  interview  you 
and  record  your  responses.  There  are  no  right  or  wrong  answers.  Your  responses  will  be 
completely  confidential  and  will  not  be  reported  to  anyone.  Based  on  data  we  extract  from 
your  record  and  your  responses,  we  can  then  provide  you  with  a  recommendation  that  may 
best  represent  your  values  and  preferences  regarding  which  approach  (targeted  treatment  or 
hospice)  you  want  to  pursue. 


Question:  Would  you  like  to  participate?  Before  we  begin,  I  will  need  to  review  this  consent 
form  with  you. 


Review  the  lnfoijned_  Consent  form  with  the  patient  Allow  the  patient  as  much  time  as  s/he 
needs  to  sign  it 

Great.  Let  me  ask  you  two  questions  before  I  go  and  review  your  medical  records. 

Question:  When  were  you  diagnosed  with  this  condition? 

Record  onswer_  on  Checklist 

Question:  May  I  also  have  your  phone  number  so  that  we  can  follow  up  with  you? 

Record  answer  on  Checklist 


Thanlj  the  patient  and_  discuss  when  would_  be  a  convenient  time  for  you  to  return  to  conduct  the 
interview. 
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Patient  Script  to  Accompany  the  Hospice  Referral  Software 

Enter  data  under  Interviewee  and_  Prognostic  Models  Tabs  prior  to  returning  to  ggtienfs  morn. 
Software  Tab :  Current  Choice 

Thanks  for  taking  part  in  our  study.  I  would  like  to  have  an  honest  discussion  about  the 
choice(s)  you  face.  This  will  include  finding  out  your  wishes,  values  and  preferences  about  how 
you  want  to  approach  your  illness.  This  will  also  include,  if  you  wish  to  know,  a  conversation 
with  you  about  how  long  you  may  live.  Some  people  find  talking  about  this  topic  to  be  difficult 
&  distressing.  If  at  any  point  during  our  conversation  you  feel  upset  and  wish  to  stop,  please 
don't  hesitate  to  say  so.  Your  decision  to  participate  or  not  in  this  interview  will  not  affect  the 
care  that  you  receive. 

Question:  Do  I  have  your  permission  to  proceed  with  the  interview? 

Exercise  maximum  tact,  compassion  &  imderstgnding.  Ifthe  patient  doesn't  understand,  please 
reword  as  appropriate.  Ifjn  doubts  stop  the  interview  and_  call  the  study,  coordinator/princinal 
investigator.  Proceed  only,  ifthe  oatient/surrogate  agrees  to  continue  with  the  interview. 
Otherwise  thgnl<  the  patient  &  conclude  the  interview. 

The  purpose  of  this  study  is  to  help  people  make  decisions  about  continuing  their  medical 
treatment  or  choosing  hospice  care.  As  I  mentioned,  the  main  goal  of  hospice  care  is  to  improve 
the  quality  of  patients'  lives  by  focusing  on  pain  and  other  symptom  relief  and  emotional 
comfort.  This  means  that  treatments  that  are  given  to  target  your  disease  may  be  discontinued. 

Question:  If  you  had  to  choose  right  now,  would  you  rather: 

a)  Continue  with  your  current  treatment  that  is  focused  on  targeting  your  disease,  even 
though  you  may  experience  some  pain  and  discomfort,  or, 

b)  Choose  hospice  care  that  is  focused  on  relieving  your  pain  and  discomfort  as  much  as 
possible,  even  though  it  may  not  affect  your  disease  directly,  or, 

c)  You  don't  know  at  this  point. 

As  you  arrived  at  your  decisions  and  answered  these  questions  you  might  have  thought  about 
many  different  things.  One  thing  that  may  have  occurred  to  you  is,  "Have  I  made  the  right 
choice?"  As  you  know,  we  all  make  mistakes  or  errors  from  time  to  time.  Some  errors  we  can 
accept  while  others  we  may  regret.  In  the  case  of  choosing  to  continue  your  current  treatment 
or  hospice  care,  it  is  possible  to  make  an  error.  One  way  to  study  difficult  decisions  is  to 
consider  how  much  regret  a  person  feels  if  s/he  found  out  that  s/he  had  made  the  wrong 
choice.  In  a  moment  I  will  ask  you  about  two  possible  scenarios  and  I  want  you  to  tell  me  about 
your  feelings  of  regret. 

Softwgre  Tab:  Regret 

Scenario  1:  Now  I  want  you  to  imagine  that  you  chose  to  continue  your  current  treatment  but 
six  months  from  now  you  feel  you  should  have  chosen  hospice  care  (that  is,  care  focused  on 
comfort).  How  much  regret  do  you  think  you  would  feel  in  this  situation? 
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Please  indicate  how  much  regret  you  would  feel  using  the  following  rating  scale  where  0 
indicates  no  regret  and  100  indicates  the  maximum  regret  you  can  imagine. 

Let  the  patient  use  the  arrow  keys  to  move  the  slider,  indicating  his/her  level  ofiegret 

OK,  thank  you.  Now  let's  flip  the  last  question  around. 

Scenario  2:  Imagine  that  you  chose  hospice  care  (that  is,  care  focused  on  comfort)  but  six 
months  from  now  you  feel  you  should  have  continued  your  current  treatment.  How  much 
regret  do  you  think  you  would  feel  in  this  situation? 

Please  indicate  how  much  regret  you  would  feel  using  the  following  rating  scale  where  0 
indicates  no  regret  and  100  indicates  the  maximum  regret  you  can  imagine. 

Let  the  patient  use  the  arrow  keys  to  rnove  the  slider  indicating  his/her  level  of  regret. 

Do  you  have  any  questions  about  this?  Thank  you  for  your  help.  We  are  almost  done  here.  The 
choices  that  you  just  made  will  help  us  to  come  up  with  our  best  advice  for  patients  like  you 
about  whether  to  continue  current  treatment  or  choose  hospice  care.  We  are  now  ready  to 
share  this  information  with  you. 

Question:  Would  you  like  to  know  what  appears  to  be  the  choice  about  further  treatment 
that  is  consistent  with  your  values  and  preferences? 

Reac[  thl  Summary  at  the  bottom  only,  ifthe  patient/surrogate  gives  permission.  Otherwise , 
thank  the  patient  and  clicks  "Next. " 

Please  understand  that  for  us  to  make  this  recommendation  we  needed  to  calculate  how  long  a 
patient  like  you  may  live.  This  is  our  best  (guess)  estimate  of  life  expectancy  for  an  "average" 
patient  like  you.  Some  people  live  longer  that  the  estimate  and  others  shorter.  It  is  impossible 
for  us  to  know  if  our  estimate  will  be  accurate  in  your  specific  case.  Do  you  want  to  have  a 
discussion  about  how  long  a  patient  like  you  might  live? 

Question:  Would  you  like  me  to  show  you  our  best  guess  for  how  long  a  patient  like  you 
might  live? 

Proceed_  onlyjfthe  patient/surrogate  gives  permission.  Otherwise ,  thank  the  patient  and,  clicks 
"No. "  The  software  will  skio  the  Summary  Tab  and_  go  to  the  Questionnaire  Tab. 

Software  Tabj  Summary 

Ifs/he  gives  permission^  click  "Yes, "  share  results  displayed  on  the  screen ,  and,  on  "Show  Chart. " 
Reac[  the,  exact  wording  displayed  on  the  screen,  e.g.  "Our  best  estimate  is  that  there  is  a  XX% 
chance  that  average  patients  like  you  would  live  for  6  months.  That  is,  out  of  100  patients  like 
you,  XX  will  still  be  alive  after  6  months.  This  also  means  on  average  that  (100-XX)  patients  like 
you  will  die  within  the  next  6  months.  Hospice  care  is  considered  to  be  better  for  these  (100-XX) 
patients  who  are  likely  to  die  within  the  next  6  months." 

Take  a  pause.  Give  some  time  to  the  patient  to  reflect  on  the  situation.  Exercise  maximum  tact, 
compassion  and_  understanding. 

Software  Tab:  Questionnaire 
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Question:  Thinking  back  on  the  choices  you  just  made  in  the  2  scenarios  (using  the  sliders), 
would  you  like  to  change  your  mind  (your  choices)? 

Record  the  patient' s/surrogate's  answer.  Ifthe  patient  decides  to  change  his/her  choices ,  repeat 
the  interview. 

Question:  Did  you  find  this  information  helpful? 

Yes?  No?  Somewhat? 

Question:  If  your  answer  is  "yes"  or  "somewhat,"  please  indicate  how  strongly  these 
calculations  will  influence  your  decision  for  considering  hospice  referral 

We  have  four  possible  responses:  Strongly?  Moderately  strong?  Somewhat?  or  Not  at  all? 

Question:  How  strongly  do  you  agree  with  our  best  advice  regarding  continuing  your  current 
treatment  vs.  choosing  hospice  care? 

We  have  7  possible  responses:  Strongly  agree?  Moderately  agree?  Somewhat  agree?  Neither 
agree  nor  disagree?  Somewhat  disagree?  Moderately  disagree?  or  Strongly  disagree? 

Question:  Based  on  what  we  have  talked  about  today  would  you  choose  hospice  care? 

We  have  5  possible  responses:  Yes?  No?  I  need  some  more  time  to  think  this  over?  I  need  some 
more  time  to  discuss  this  with  my  family?  or  I  need  some  more  time  to  discuss  this  with  my 
physician? 

Software  Tgbj  Acceptable  Regret 

We  will  now  continue  with  the  last  part  of  the  interview  if  you  wish  to  continue  to  participate. 
Sometimes  the  best  way  to  clarify  our  own  wishes,  values  and  preferences  is  to  imagine  that  we 
are  in  a  position  to  advise  other  people  who  find  themselves  in  the  situation  in  which  we  find 
ourselves.  There  are  two  questions. 

First,  we  would  like  to  see  under  which  circumstances  you  would  not  regret  making  a  wrong 
decision.  Imagine  that  there  are  100  patients  like  you  who  will  live  as  long  as  you  will.  Pretend 
that  you  are  a  doctor  and  you  need  to  decide  whether  these  patients  will  go  to  hospice  care  or 
they  will  continue  their  current  treatment.  This  is  akin  as  asking  your  doctor,  "What  would  you 
do  if  you  were  me?"  Please  remember,  there  is  no  guarantee  that  your  decision  will  be 
accurate.  That  is,  like  all  of  us,  you  can  make  mistakes.  However,  some  people  are  better  than 
others  in  accepting  their  mistakes.  We  would  like  to  see  how  much  you  accept  making  a 
mistake  in  this  situation  without  regretting  it. 

Question:  Read  exact  wording  on  screen .  e.g.,  "So  let's  pretend  that  you  are  a  doctor.  Based  on 
our  calculations,  you  would  refer  X  patients  out  of  100  to  hospice.  It  is  likely  that  most  of  these 
patients  will  die  within  next  6  months.  Therefore,  they  are  expected  to  benefit  from  care 
through  hospice.  However,  some  of  these  patients  may  live  longer  than  6  months  and  hence 
should  not  have  been  referred  to  hospice.  As  a  doctor,  how  many  patients  (out  ofX)  would  you 
be  willing  to  incorrectly  (wrongly)  refer  to  hospice  and  still  feel  no  regret?" 

Thank  you.  There  is  only  one  more  question.  Let's  flip  the  last  question  around. 
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Question:  Read_  exact  wordinci  on  screen  e.g.,  "Based  on  our  calculations,  you  would 
recommend  X  patients  out  of  100  to  continue  current  treatment.  It  is  likely  that  most  of  these 
patients  will  live  longer  than  6  months.  Therefore,  they  are  expected  to  benefit  from  the  current 
treatment.  However,  some  of  these  patients  may  live  less  than  6  months  and  hence  should  have 
been  referred  to  hospice.  As  a  doctor,  how  many  patients  (out  ofX)  would  you  be  willing  to 
incorrectly  (wrongly)  keep  on  current  treatment  and  still  feel  no  regret?" 

We  truly  appreciate  your  willingness  to  participate.  Do  you  have  any  questions? 

Answer  grvy_  questions  you  can  and/or  suggest  the  patient  speak  with  their,  physician  or  hospice 
representative. 

I  hope  your  participation  has  been  helpful  to  you.  We  truly  appreciate  it.  If  it's  still  OK  with  you, 
we  would  like  to  contact  you  every  month  or  so  to  ask  you  a  few  follow  up  questions.  Thank 
you  again. 

Software  Talx  Final 

Save  the  file. 
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Introductory  Script 

Good  morning/afternoon.  My  name  is _ .  I'm  working  with  Dr.  Ben 

Djulbegovic,  a  Moffitt  oncologist  and  USF  Researcher.  We  would  like  to  tell  you  about  our 
Hospice  Decision  Support  research  study  that  we  think  can  help  you  with  current  and  future 
patients.  As  I'm  sure  you  know,  it's  widely  believed  that  patients  are  generally  referred  to 
hospice  much  later  than  they  should  be.  Our  decision  support  tool  helps  in  two  ways:  It 
improves  upon  existing  prognostication  methods  for  determining  terminal  patients'  remaining 
life  spans  and  uses  patients'  values  and  preferences  to  help  physicians  better  guide  patients  in 
choosing  whether  to  choose  targeted  treatment  of  their  disease  or  hospice  care. 

You  recently  referred  a  patient  to  us:  Mr/s. _ in  Rm. _ .  S/he  was 

a _ diagnosed  with _ . 

Question:  Do  you  remember  the  patient?  IF  NO,  thank  him/her  for  the  referral  and  his/her 
time  today,  and  say  that  we'll  try  to  catch  him  sooner  on  a  future  case. 

We  interview  not  only  the  patient  but  when  possible  their  physician.  The  physician  interview 
only  takes  about  10  minutes.  There  are  no  right  or  wrong  answers.  Your  responses  will  be 
completely  confidential  and  will  not  be  reported  to  anyone.  Based  on  data  we  extract  from  the 
patient's  record  and  your  responses,  we  can  then  provide  you  with  feedback  regarding  this 
particular  patient  that  you  may  find  helpful. 

Question:  Would  you  like  to  participate?  Are  you  available  now  or  should  we  set  up  another 
time?  I  will  need  to  review  this  informed  consent  form  with  you  and  get  your  signature. 

Review  the  Informed  Consent  form  with  the  physician.  Allow  him/her  as  much  time  as  s/he 
needs  to  sign  it 
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Physician  Script  to  Accompany  the  Hospice  Referral  Software 

Software  Tab:  Interviewee 

Enter  patients  randomly  assigned  number  and_  select  "Physician. " 

Click  through  Prognostic  Models  Tab. 

Software  Tab:  Current  Choice 

Thanks  for  taking  part  in  our  study.  I  would  like  to  have  a  frank  discussion  with  you  about 
methods  for  estimating  life  expectancy  in  terminally  ill  patients.  If  you  find  talking  about  this 
topic  to  be  difficult  and  distressing  at  any  point  during  our  conversation  and  wish  to  stop,  just 
say  so. 

Question:  Do  I  have  your  permission  to  proceed  with  the  interview? 

Offer  to  give  the  physician  study,  rnaterial  to  read  first.  Proceed_  only,  if  the  physician  agrees  to 
continue  with  the  interview.  Otherwise  thank  the  physician  and_  conclude  the  interview. 

The  purpose  of  this  study  is  to  find  out  how  terminally  ill  people  make  decisions  about  choosing 
targeted  treatment  or  hospice.  We  also  want  to  study  how  physicians  make  decisions  regarding 
recommending  that  their  terminally  ill  patients  choose  targeted  treatment  or  care  through 
hospice.  As  you  know,  either  choice  may  or  may  not  prolong  patients'  lives.  The  main  goal  of 
hospice  care  is  to  improve  quality  of  patient's  life  by  focusing  on  pain  and  other  symptom  relief 
and  emotional  comfort.  This  means  that  treatment(s)  that  might  extend  a  patient's  life  may  be 
discontinued. 

Question:  If  you  had  to  choose  right  now,  would  you  rather: 

a)  Continue  with  current  treatment  that  is  focused  on  extending  your  patient's  life  as  much  as 
possible,  even  though  s/he  may  experience  some  pain  and  discomfort,  or, 

b)  Choose  hospice  care  that  is  focused  on  relieving  your  patient's  pain  and  discomfort  as 
much  as  possible,  even  though  s/he  might  not  live  as  long,  or 

c)  You  don't  know  at  this  point. 

As  you  arrived  at  your  decisions  and  answered  these  questions  you  might  have  thought  about 
many  different  things.  One  thing  that  may  have  occurred  to  you  is,  "Have  I  made  the  right 
choice?"  As  you  know,  we  all  make  mistakes  or  errors  from  time  to  time.  Some  errors  we  can 
accept  while  others  we  may  regret.  In  the  case  of  recommending  continued  treatment  or 
hospice  care  for  your  patient,  it  is  possible  to  make  an  error. 

One  way  to  study  difficult  decisions  is  to  consider  how  much  regret  a  person  would  feel  if  s/he 
found  out  that  s/he  had  made  the  wrong  choice.  In  a  moment  I  will  ask  you  about  two  possible 
scenarios  and  I  want  you  to  tell  me  about  your  feelings  of  regret. 

Software  Tab:  Regret 
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Scenario  1:  Now,  I  want  you  to  imagine  that  you  chose  to  continue  your  current  treatment  for 
your  patient  but  six  months  from  now  you  feel  you  should  have  chosen  hospice  care  (care 
focused  on  comfort).  How  much  regret  do  you  think  you  would  feel  in  this  situation? 

Please  indicate  how  much  regret  you  would  feel  using  the  following  rating  scale  where  0 
indicates  no  regret  and  100  indicates  the  maximum  regret  you  can  imagine. 

Let  the  physician  move  the  slider  or  use  the  arrow  keys  to  indicate  his/her  level  °f  regret. 

OK,  thank  you.  Now  let's  flip  the  last  question  around. 

Scenario  2:  Imagine  that  you  chose  hospice  care  (care  focused  on  comfort)  but  six  months 
from  now  you  feel  you  should  have  continued  current  treatment  for  your  patient.  How  much 
regret  do  you  think  you  would  feel  in  this  situation? 

Please  indicate  how  much  regret  you  would  feel  using  the  following  rating  scale  where  0 
indicates  no  regret  and  100  indicates  the  maximum  regret  you  can  imagine. 

Let  the  /physician  rnove  the  slider.  or_  use  the  arrow,  keys  to  indicate  his/her  level  of  regret. 

Do  you  have  any  questions  about  this?  Thank  you  for  your  help.  We  are  almost  done  here.  The 
choices  that  you  just  made  will  help  us  to  come  up  with  our  best  advice  for  patients  like  the  one 
you  are  treating  about  whether  to  continue  current  treatment  or  choose  hospice  care.  We  are 
now  ready  to  share  this  information  with  you. 

Question:  Would  you  like  to  know  what  appears  to  be  the  choice  consistent  with  your  values 
and  preferences  about  further  treatment? 

Reac[  the  Summary  at  the  bottom  only,  if  the  physician  gives  permission.  Otherwise ,  thank  the 
physician  and,  clicks  "Next. " 

Please  understand  that  for  us  to  determine  whether  you  should  advise  your  patient  to  continue 
his  current  treatment  or  choose  hospice  care  we  needed  to  calculate  how  long  a  patient  like 
the  one  you  are  treating  may  live.  This  is  our  best  (guess)  estimate  for  an  average  patient  like 
him/her.  As  you  know,  some  people  live  longer  that  the  estimate  and  others  shorter.  It  is 
impossible  for  us  to  know  if  our  estimate  will  be  accurate  in  your  patient's  case. 

Question:  Would  you  like  me  to  show  you  our  best  guess  for  how  long  a  patient  like  the  one 
you  are  treating  might  live? 

Proceed  onlyjfthe  [physician  gives  permission.  Otherwise  thank  the  physician  and_  clicks  "No. " 
The  software  wilj  skip  the  Summary,  Tab  and_  go  to  the  Questionnaire  Tab. 

Software  Tab :  Summary 

Ifs/he  gives  permission^  click  "Yes, "  share  results  displayed  on  the  screen  and  on  " Show  Chart. " 
Read_  the  exact  wording  displayed  on  the  screen ,  e.g.  " Our  best  estimate  is  that  there  is  XX% 
chance  that  an  average  patient  like  the  one  you  are  treating  will  live  for  6  months.  That  is,  out 
of  100  patients  like  the  one  you  are  treating,  XX  will  still  be  alive  after  6  months.  This  also  means 
on  average  that  (100-XX)  patients  like  him/her  will  die  within  the  next  6  months.  Hospice  care  is 
considered  to  be  better  for  these  (100-XX)  patients  who  are  likely  to  die  within  the  next  6 
months." 
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Take  a  pause.  Give  the  physician  time  to  reflect  on  the  situation.  Exercise  maximum  tact, 
compassion  and  understanding. 

Use  both  the  case  report  forms  and,  the  Hospice  Referral  software  to  record  the  physician's 
answers. 

Software  Tab :  Questionnaire 

Question:  Thinking  back  on  the  choices  you  just  made  in  the  2  scenarios  (using  the  sliders), 
would  you  like  to  change  your  mind  (your  choices)? 

Record  the  ghysician^s  answer  using  both  the  case  report  forms  and,  the  Hospice  Referral 
Software .  If  the  s/he  decides  to  change  his/her  choices ,  repeat  the  interview. 

Question:  Did  you  find  this  information  helpful? 

Yes?  No?  Somewhat? 

Question:  If  your  answer  is  "yes"  or  "somewhat,"  please  indicate  how  strongly  this 
information  will  influence  your  decision  for  considering  hospice  referral  for  your  patient. 

We  have  four  possible  responses:  "Strongly"?  "Moderately  strongly"?  "Somewhat"?  or  "Not  at 
all"? 

Question:  How  strongly  do  you  agree  with  our  best  advice  regarding  continuing  current 
treatment  vs.  choosing  hospice  care  for  your  patient? 

We  have  7  possible  responses:  Strongly  agree?  Moderately  agree?  Somewhat  agree?  Neither 
agree  nor  disagree?  Somewhat  disagree?  Moderately  disagree?  or  Strongly  disagree? 

Question:  Based  on  what  we  have  talked  about  today  would  you  choose  hospice  care  for  your 
patient? 

We  have  5  possible  responses:  Yes?  No?  I  need  some  more  time  to  think  this  over?  I  need  some 
more  time  to  discuss  this  with  the  family?  or  I  need  some  more  time  to  discuss  this  with 
another  physician? 
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Hospice  Decision  Support  Physician  Scripts 


Software  [a]x  Acceptable  Regret 

We  have  just  two  more  questions.  We  would  like  to  see  under  which  circumstances  you  would 
not  regret  if  you  made  a  wrong  decision.  Imagine  that  you  are  treating  100  terminally  ill 
patients.  You  need  to  decide  whether  these  patients  will  go  to  hospice  care  or  they  will 
continue  treatment.  Please  remember,  there  is  no  guarantee  that  your  decision  will  be 
accurate.  That  is,  like  all  of  us  you  can  make  mistakes.  However,  some  people  are  better  than 
others  in  accepting  their  mistakes.  We  would  like  to  see  how  much  you  accept  making  a 
mistake  in  this  situation  without  regretting  it. 

Question:  Read,  exact  wording  on  screen .  e.g.,  "Based  on  our  calculations,  you  would  refer  X 
patients  out  of  100  to  hospice.  It  is  likely  that  most  of  these  patients  will  die  within  the  next  6 
months.  Therefore,  they  are  expected  to  benefit  from  hospice  care.  However,  some  of  these 
patients  may  live  longer  than  6  months  and  hence  should  not  have  been  referred  to  hospice. 
How  many  patients  (out  of  X)  would  you  be  willing  to  incorrectly  (wrongly)  refer  to  hospice  and 
still  feel  no  regret? 

Record  the  participating,  physician's  answer.  Take  a  pause. 

OK,  just  one  more  question.  Let's  flip  that  last  question  around. 

Question:  Read,  exact  wording  on  screen .  e.g.  "Based  on  our  calculations,  you  would 
recommend  X  patients  like  the  patient  you  are  treating  out  of  100  continue  their  current 
treatment.  It  is  likely  that  most  of  these  patients  will  live  longer  than  6  months.  Therefore,  they 
are  expected  to  benefit  from  the  current  treatment.  However,  some  of  these  patients  may  live 
less  than  6  months  and  hence  should  have  been  referred  to  hospice.  How  many  patients  (out  of 
X)  would  you  be  willing  to  incorrectly  (wrongly)  keep  on  current  treatment  and  still  feel  no 
regret ?" 

Record  the  participating  physician's  answer. 

I  hope  this  information  has  been  helpful  to  you.  We  truly  appreciate  your  willingness  to 
participate.  Do  you  have  any  questions? 

Answer  any,  guestions  you  can  and/or  refer  the  physician  to  Dr.  Diulbeaovic. 

Thank  you. 

Software  Talx  Final 
Save  the  file. 
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Appendix  10  Informed  consent  forms 


USF 


UNIVERSITY  OF 
SOUTH  FLORIDA 

Informed  Consent  to  Participate  in  Research  and 
Authorization  to  Collect,  Use  and  Share  Your  Health  Information 

For  Patient 

Information  to  Consider  Before  Taking  Part  in  this  Research  Study 
IRB  Study  #  Pro00000220 


Researchers  at  the  University  of  South  Florida  (USF),  Moffitt  Cancer  Center  and  Tampa  General 
Hospital  (TGH)  study  many  topics.  To  do  this,  we  need  the  help  of  people  who  agree  to  take  part  in  a 
research  study.  This  form  tells  you  about  this  research  study. 

We  are  asking  you  to  take  part  in  a  research  study  that  is  called: 

“Proposal  for  development  of  evidence  based  clinical  decision  support  system  (EBM-CDSS)  to  aid 
prognostication  in  terminally  ill  patients” 

We  are  approaching  you  to  ask  you  to  participate  in  this  study  because  we  were  told  that  you  have  a 
terminal  disease  that  may  significantly  shorten  your  life  duration.  You  may  participate  in  this  study; 
only  if  you  aware  that  you  have  a  terminal  disease. 

The  person  who  is  in  charge  of  this  research  study  is  Dr.  Benjamin  Djulbegovic.  This  person  is  called 
the  Principal  Investigator.  However,  other  research  staff  may  be  involved  and  can  act  on  behalf  of  the 
person  in  charge. 

The  person  explaining  the  research  to  you  may  be  someone  other  than  the  Principal  Investigator. 

Other  research  personnel  who  you  may  be  involved  with  include:  your  consulting  physician  and  or  the 
following  study  personnel:  Catherine  Jahrsdorfer,  Mario  Crawford,  Rahul  Mhaskar  and  Howard  Tuch. 


The  research  will  be  done  at  Tampa  General  Hospital,  Moffitt  Cancer  Center  and  Tampa  Bay  Lifepath 
Hospice  and  Palliative  Care  and  Moffitt  cancer  center. 


This  research  is  being  paid  for  by  the  United  States  Army. 


Purpose  of  the  study 

The  purpose  of  this  study  is  to  help  people  in  the  final  stages  of  their  lives  make  decisions  about 
continuing  medical  treatment  targeting  their  disease  or  choosing  hospice  care.  Hospice  is  a  type  of  care 
that  provides  support  to  people  at  the  end  of  their  lives.  The  main  goal  of  hospice  care  is  to  improve 
the  quality  of  patients’  lives  by  focusing  on  pain  and  other  symptom  relief  and  emotional  comfort.  This 
means  that  medical  treatments  that  are  given  to  target  your  disease  may  be  discontinued. 


IRB  Number:  Pro00000220  Version  3  IRB  Consent  Rev.  Date:  23  Aug  2012 _ 

IC  Adult  Minimal  Risk  RA  Template  -  SB  Rev:  9-3-2010  Page  1  of  9 


This  research  study  is  aimed  at  helping  you  choose  what  type  of  medical  care  you  want  to  have  during 
the  upcoming  months.  Most  people  in  this  situation  are  uncertain  how  they  want  to  approach  their 
illness.  We  have  created  a  tool  to  help  people  and  their  doctors  make  the  choice  that  they  feel  is  right 
for  them.  The  tool  is  designed  to  help  doctors  and  patients  like  yourself  decide  whether  to  continue 
treatment  targeting  their  disease  although  the  treatment  may  create  additional  discomfort,  or  whether  to 
choose  care  through  hospice  which  is  focused  on  providing  comfort  and  symptom  relief.  Either  of 
these  choices  may  or  may  not  prolong  your  life.  We  would  also  like  to  understand  your  wishes  and 
preferences  to  help  you  make  the  choice  that  is  right  for  you. 


The  tool  that  is  being  tested  in  this  research  is  called  an  Evidence-based  Medicine  Clinical  Decision 
Support  System  (EBM-CDSS).  The  EBM-CDSS  is  designed  to  be  used  by  doctors  at  the  bedside  to 
better  predict  life  expectancy  in  patients  with  terminal  illness  and  to  improve  the  timing  and 
appropriateness  of  referral  to  hospice  care. 


We  are  asking  you  to  take  part  in  this  research  because  you  have  an  illness  that  may  significantly 
shorten  your  life  span. 


How  many  other  people  will  take  part? 

About  400  people  will  take  part  in  this  study. 


Study  Procedures 

If  you  take  part  in  this  study:  We  will  actively  collect  information  from  you  until  your  demise.  The 
study  personnel  or  your  attending  physician  will  visit  you  twice  initially  and  then  once  a  month  during 
the  study  period  and  will  collect  information  from  you.  We  do  not  expect  the  initial  visits  to  last  more 
than  30  minutes  and  the  monthly  visits  to  last  more  than  10  minutes. 

If  you  take  part  in  this  study,  you  will  be  requested  to  sign  this  informed  consent  form: 

•  After  signing  the  informed  consent  you  will  be  requested  by  the  study  personnel  to  share 
certain  information  related  to  your  health  from  your  medical  records.  You  may  be  requested  to 
answer  questions  regarding  your  health. 

•  The  interviewer  will  also  ask  you  whether  you  would  like  to  know  your  doctor’s  best  estimate 
of  your  life  expectancy. 

•  We  will  estimate  your  life  expectancy  based  on  our  current  best  estimates  for  “average” 
patients  like  you  (i.e.  estimate  of  the  average  future  lifetime  for  people  like  you)?  We  will  ask 
you  whether  you  would  like  to  know  the  best  estimates  of  your  life  expectancy  with  the  current 
treatment  targeted  at  your  disease  and  without  this  treatment. 

•  To  help  you  decide  between  hospice  care  and  treatment  targeted  at  your  disease.  We  will  also 
like  to  learn  about  your  wishes  and  preferences  regarding  how  you  would  like  to  approach  your 
illness.  We  will  do  that  by  using  our  tool  to  understand  your  feelings  of  regret  in  case  you  made 
a  wrong  choice. 

•  The  interviewer  will  also  seek  your  opinion  regarding  your  tolerance  if  your  doctor  is  mistaken 
regarding  whether  to  continue  your  present  treatment  (that  may  potentially  cure  your  disease  or 
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prolong  your  life)  or  to  discontinue  your  treatment  and  refer  you  to  hospice  care.  We  will  ask 
you  a  series  of  questions  assuming  that  there  is  an  error  in  prediction  of  your  life  expectancy 
with  and  without  treatment  targeted  at  your  disease.  We  want  to  know  how  large  an  error  you 
can  tolerate  and  still  not  regret  (magnitude  of  acceptable  regret  for  you)  your  doctor’s  decision 
to  be  referred  to  hospice  care.  After  you  make  the  decision  regarding  choosing  hospice  care  or 
continuing  treatment  we  will  ask  you  questions  to  obtain  your  opinion  regarding  the  decision 
you  made  once  a  month  until  you  pass  away. 

•  We  will  ask  you  whether  you  changed  your  decision  for  enrolling  /  not  enrolling  in  hospice 
care  based  on  the  information  provided  to  you  by  us. 

•  You  should  know  that  we  cannot  tell  you  which  of  these  choices  (treatment  targeted  at  your 
disease  versus  hospice  care)  will  prolong  your  life.  However,  we  may  be  able  to  tell  you  which 
of  these  choices  is  most  compatible  with  your  wishes  and  preferences  for  an  average  patient 
like  you. 


Alternatives 

You  do  not  have  to  take  part  in  this  study.  Your  decision  to  take  part  in  this  study  or  not  to  take  part 
will  not  affect  the  medical  care  that  you  receive  You  can  choose  to  fully  take  part  in  this  study 
including  the  collection  of  your  medical  record  information  and  the  initial  and  follow  up  interviews. 

You  can  also  choose  to  partially  participate  in  this  study  either  by: 

•  allowing  us  to  look  at  your  medical  records  only  and  NOT  take  part  in  the  interview  process 
OR 

•  NOT  allow  us  to  look  at  your  medical  records  or  interview  you  but  ONLY  allow  us  to  collect 
the  date  that  you  eventually  pass  away. . 

You  choice  of  whether  to  take  part  in  the  study  fully  or  one  of  the  partial  options 
will  be  indicated  on  the  last  pages  of  this  consent  form.  There  is  a  separate 
signature  area  for  each  choice  and  only  one  of  these  sections  should  be  signed. 
Benefits 

We  don’t  know  if  you  will  get  any  benefits  by  taking  part  in  this  study.  However,  this  study  may 
improve  doctor’s  accuracy  to  assess  life  expectancy.  Many  people  find  this  important,  which  can  be 
important  for  you  too.  Many  people  also  find  it  helpful  to  clarify  their  wishes  and  preferences 
regarding  how  they  would  like  to  spend  the  reminder  of  their  life. 


Risks  or  Discomfort 

This  research  is  considered  to  be  minimal  risk.  That  means  that  the  risks  associated  with  this  study  are 
the  same  as  what  you  face  in  the  current  phase  of  your  life  on  an  everyday  basis.  There  are  no  known 
additional  risks  to  those  who  take  part  in  this  study. 

While  many  people  in  your  situations  want  to  have  a  frank  conversation  about  doctors’  best  estimate  of 
their  life  expectancy,  we  understand  that  talking  about  how  to  spend  your  remaining  days  of  life  is 
difficult  and  can  be  very  distressing.  If  you  find  anything  that  we  are  about  to  ask  you  upsetting  and 
distressing,  you  may  request  to  discontinue  your  participation  from  the  study.  There  is  no  funding  to 
compensate  you  for  treatment  if  “injury”  occurs.  There  might  be  other  unforeseeable  risks  that  could 
occur  which  are  unknown  at  this  time.  Your  decision  to  withdraw  from  the  study  will  not  affect  the 
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care  you  will  receive  in  any  form  and  or  manner.  If  you  wish  to  consult  the  study  Principal  Investigator 
and  Co-  Principal  Investigator  (who  have  extensive  experience  working  with  patients  in  similar 
situations)  will  immediately  be  available  for  counseling. 


Compensation 

You  will  receive  no  payment  or  other  compensation  for  taking  part  in  this  study. 


Cost 

There  will  be  no  additional  costs  to  you  as  a  result  of  being  in  this  study.  However,  routine  medical 
care  for  your  condition  (care  you  would  have  received  whether  or  not  you  were  in  this  study)  will  be 
charged  to  you  or  your  insurance  company. 


Conflict  of  Interest  Statement 

There  is  no  conflict  of  interest. 


Authorization  to  Use  and  Disclose  Protected  Health  Information 

Who  will  see  your  health  information  ? 

In  our  research,  we  use  and  share  your  health  information  to  the  extent  authorized  by  you.  We  know 
that  this  information  is  private.  The  federal  privacy  regulations  of  the  Health  Insurance  Portability  & 
Accountability  Act  (HIPAA)  protect  your  identifiable  health  information.  If  you  authorize  us  to  use 
your  information  we  will  protect  it  as  required  by  the  law. 

Research  at  Tampa  General  Hospital  and  Moffitt  cancer  center  is  conducted  jointly  with  the  University 
of  South  Florida.  By  signing  this  form,  you  are  permitting  Tampa  General  Hospital,  Moffitt  cancer 
center  and  the  University  of  South  Florida  to  use  personal  health  information  collected  about  you  for 
research  purposes.  You  are  also  allowing  Tampa  General  Hospital  and  Moffitt  cancer  center  to  share 
your  personal  health  information  with  individuals  or  organizations  other  than  USF,  Moffitt  cancer 
center  and  Tampa  General  Hospital  who  are  also  involved  in  this  research  and  listed  below. 


Who  will  disclose  (share),  receive,  and/or  use  your  information? 

To  do  this  research,  USF  and  the  people  and  organizations  listed  below  may  use  or  share  your 
information.  They  may  only  use  and  share  your  information: 

•  With  the  people  and  organizations  on  this  list; 

•  With  you  or  your  personal  representative;  and 

•  As  allowed  by  law. 

In  addition  to  the  people  and  organizations  listed  below  in  the  Privacy  and  Confidentiality  section  of 
this  document,  the  following  groups  of  people  may  also  be  able  to  see  information  about  you  and  may 
use  the  information  to  conduct  the  research: 
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•  The  medical  staff  that  takes  care  of  you  and  those  who  are  part  of  this  research  study; 

•  Each  research  site  for  this  study.  This  includes  the  research  and  medical  staff  at  each  site  and 
USF; 

•  The  designated  peer  review  committees  such  as  the  Tampa  General  Hospital  and  Moffitt  cancer 
center  Feasibility  Committee; 

•  Additionally,  there  may  be  other  people  and/or  organizations  who  may  be  given  access  to  your 
personal  health  information.  This  includes  Tampa  Bay  LifePath  Hospice  care. 


Who  else  can  use  and  share  this  information? 

Anyone  listed  above  may  use  consultants  in  this  research,  and  for  the  purpose  of  this  study  may  share 
your  information  with  them.  If  you  have  questions  about  who  they  are,  you  can  ask  us.  Individuals 
who  receive  your  health  information  for  this  research  study  may  not  be  required  by  the  HIPAA  Privacy 
Rule  to  protect  it  and  may  share  your  information  with  others  without  your  permission.  They  can  do  so 
if  permitted  by  the  laws  governing  them.  Example:  The  sponsor  may  share  your  information.  If  the 
sponsor  or  others  share  your  information,  your  information  may  no  longer  be  protected  under  the 
HIPAA  Privacy  Rule. 

How  Will  My  Information  Be  Used? 

By  signing  this  form,  you  are  giving  your  permission  to  use  and/or  share  your  health  information  as 
described  in  this  document  for  any  and  all  study/research  related  purposes.  Your  authorization 
(permission)  to  use  your  health  information  will  not  expire  until  the  end  of  this  research  study  unless 
you  revoke  this  authorization  in  writing. 

As  part  of  this  research,  USF  may  collect,  use,  and  share  the  following  information: 

•  Your  whole  research  record 

•  All  of  your  past,  current  or  future  medical  and  other  health  records  held  by  USF,  other  health 
care  providers  or  any  other  site  affiliated  with  this  study.  This  includes,  but  is  not  limited  to, 
HIV/AIDs,  mental  health,  substance  abuse,  and/or  genetic  information. 

We  may  publish  what  we  learn  from  this  study.  If  we  do,  we  will  not  let  anyone  know  your  name.  We 
will  not  publish  anything  else  that  would  let  people  know  who  you  are. 

Your  Rights: 

You  can  refuse  to  sign  this  form.  If  you  do  not  sign  this  form: 

•  You  will  not  be  able  to  take  part  in  this  research  and  therefore  not  be  able  to  receive  the 
research  related  interventions.  However,  you  can  receive  other  treatments  that  are  currently 
available  for  you  as  part  of  your  regular  medical  treatment. 

•  This  will  not  change  your  health  care  outside  of  this  study. 

•  This  will  not  change  your  health  care  benefits. 

•  This  will  not  change  the  costs  of  your  health  care. 


How  Do  I  Withdraw  Permission  to  Use  My  Information? 

You  can  revoke  this  form  at  any  time  by  sending  a  signed  letter  to  USF  at  the  address  given  below, 
clearly  stating  that  you  wish  to  withdraw  your  authorization  to  the  use  of  your  health  information  in  the 
research.  If  you  revoke  you  permission: 
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•  You  will  no  longer  be  a  participant  in  this  research  study. 

•  We  will  stop  collecting  new  information  about  you. 

•  We  will  use  the  information  collected  prior  to  the  revocation  of  your  authorization.  This 
information  may  already  have  been  used  or  shared  with  others,  or  we  may  need  it  to  complete 
and  protect  the  validity  of  the  research.. 

•  Staff  may  follow-up  with  you  if  there  is  a  medical  reason  to  do  so. 

To  revoke  this  form,  you  must  tell  us  in  writing.  Please  write  to: 

Principal  Investigator:  Dr.  Benjamin  Djulbegovic 
For  IRB  Study  #  Pro00000220 
12901  Bruce  B.  Downs  Blvd.,  MDC27 
Tampa  FL,  33612 


While  we  are  doing  this  research,  we  cannot  let  you  see  or  copy  the  research  information  we  have 
about  you.  After  the  research  is  done,  you  or  a  person  designated  by  you  have  a  right  to  see  the 
information  about  you,  as  allowed  by  USF  policies. 

Privacy  and  Confidentiality 

We  will  keep  your  study  records  private  and  confidential.  Certain  people  may  need  to  see  your  study 
records.  By  law,  anyone  who  looks  at  your  records  must  keep  them  completely  confidential.  The  only 
people  who  will  be  allowed  to  see  these  records  are: 

•  The  research  team,  including  the  Principal  Investigator,  study  coordinator,  research  nurses,  and 
all  other  research  staff. 

•  Certain  government  and  university  people  who  need  to  know  more  about  the  study.  For 
example,  individuals  who  provide  oversight  on  this  study  may  need  to  look  at  your  records. 
This  is  done  to  make  sure  that  we  are  doing  the  study  in  the  right  way.  They  also  need  to  make 
sure  that  we  are  protecting  your  rights  and  your  safety. 

•  Any  agency  of  the  federal,  state,  or  local  government  that  regulates  this  research.  This  includes 
the  Food  and  Drug  Administration  (FDA),  Florida  Department  of  Health,  and  the  Department 
of  Health  and  Human  Services  (DHHS)  and  the  Office  for  Human  Research  Protection 
(OHRP). 

•  The  USF  Institutional  Review  Board  (IRB)  and  its  related  staff  who  have  oversight 
responsibilities  for  this  study,  staff  in  the  USF  Office  of  Research  and  Innovation,  USF 
Division  of  Research  Integrity  and  Compliance,  and  other  USF  offices  who  oversee  this 
research. 

•  The  sponsors  of  this  study  The  United  States  Army 

We  may  publish  what  we  learn  from  this  study.  If  we  do,  we  will  not  include  your  name.  We  will  not 
publish  anything  that  would  let  people  know  who  you  are. 
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Voluntary  Participation  /  Withdrawal 

You  should  only  take  part  in  this  study  if  you  want  to  volunteer.  You  should  not  feel  that  there  is  any 
pressure  to  take  part  in  the  study,  to  please  the  investigator  or  the  research  staff.  You  are  free  to 
participate  in  this  research  or  withdraw  at  any  time.  There  will  be  no  penalty  or  loss  of  benefits  you 
are  entitled  to  receive  if  you  stop  taking  part  in  this  study. 

New  information  about  the  study 

During  the  course  of  this  study,  we  may  find  more  information  that  could  be  important  to  you.  This 
includes  information  that,  once  learned,  might  cause  you  to  change  your  mind  about  being  in  the  study. 
We  will  notify  you  as  soon  as  possible  if  such  information  becomes  available. 


You  can  get  answers  to  your  questions,  concerns,  or  complaints 

If  you  have  any  questions,  concerns  or  complaints  about  this  study,  or  experience  an  adverse  event  or 
unanticipated  problem,  call  Dr.  Benjamin  Djulbegovic  at  813-396-9178. 

If  you  have  questions  about  your  rights  as  a  participant  in  this  study,  general  questions,  or  have 
complaints,  concerns  or  issues  you  want  to  discuss  with  someone  outside  the  research,  call  the 
Division  of  Research  Integrity  and  Compliance  of  the  University  of  South  Florida  at  (813)  974-5638. 
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Consent  to  Take  Part  in  this  Research  Study  and 
Authorization  to  Collect,  Use  and  Share  Your  Health  Information 

It  is  up  to  you  to  decide  whether  you  want  to  take  part  in  this  study.  If  you  want  to  take  part,  please 
sign  the  form,  if  the  following  statements  are  true. 

Full  Consent  to  Take  Part  in  this  Research,  including  medical  record  review,  initial  interview 
and  follow  up  interviews: 


I  freely  give  my  consent  to  take  part  in  this  study  and  authorize  that  my  health  information  as 
agreed  above,  be  collected/disclosed  in  this  study.  I  understand  that  I  am  allowing  the  research  team 
to  access  my  health  records  for  the  purpose  of  this  study  and  interview  me  regarding  my  health  and  my 
decision  regarding  enrolling  to  hospice.  I  understand  that  by  signing  this  form  I  am  agreeing  to  take 
part  in  research.  I  have  received  a  copy  of  this  form  to  take  with  me. 


Signature  of  Person  Taking  Part  in  Study 


Date 


Printed  Name  of  Person  Taking  Part  in  Study 


Partial  Consent  to  Take  Part  in  this  Research,  Without  Participation  in  interview  process 

It  is  up  to  you  to  decide  whether  you  want  to  take  part  in  this  study.  If  you  want  to  take  part,  please 
sign  the  form,  if  the  following  statements  are  true. 

I  freely  give  my  consent  to  take  part  in  this  study  and  authorize  that  my  health  information  as 
agreed  above,  be  collected/disclosed  in  this  study.  I  understand  that  I  am  allowing  the  research  team 
to  access  my  health  records  for  the  purpose  of  this  study.  I  understand  that  by  signing  this  form  I  am 
agreeing  to  take  part  in  research.  I  have  received  a  copy  of  this  form  to  take  with  me. 

I  am  allowing  the  researchers  to  access  my  medical  record  and  I  am  NOT  willing  to  participate  in  the 
interview  process. 


Signature  of  Person  Taking  Part  in  Study 


Date 


Printed  Name  of  Person  Taking  Part  in  Study 


IRB  Number:  Pro00000220  Version  3  IRB  Consent  Rev.  Date:  23  Aug  2012 _ 

IC  Adult  Minimal  Risk  RA  Template  -  SB  Rev:  9-3-2010  Page  8  of  9 


Partial  Consent  to  Take  Part  in  this  Research,  Without  Participation  in  interview  process  or 

medical  record  review 


It  is  up  to  you  to  decide  whether  you  want  to  take  part  in  this  study.  If  you  want  to  take  part,  please 
sign  the  form,  if  the  following  statements  are  true.  I  freely  give  my  consent  to  take  part  in  this  study 
and  authorize  that  my  health  information  as  agreed  above,  be  collected/disclosed  in  this  study.  I 
understand  that  I  am  allowing  the  research  team  to  collect  the  date  that  I  eventually  pass  away  for  the 
purpose  of  this  study.  I  understand  that  by  signing  this  form  I  am  agreeing  to  take  part  in  research.  I 
have  received  a  copy  of  this  form  to  take  with  me. 

I  ONLY  ALLOW  researchers  to  access  medical  records  to  collect  the  date  that  I  eventually  pass  away. 


Signature  of  Person  Taking  Part  in  Study 


Date 


Printed  Name  of  Person  Taking  Part  in  Study 

Statement  of  Person  Obtaining  Informed  Consent  and  Research  Authorization 

I  have  carefully  explained  to  the  person  taking  part  in  the  study  what  he  or  she  can  expect  from  their 
participation. 

I  hereby  certify  that  when  this  person  signs  this  form,  to  the  best  of  my  knowledge,  he  or  she 
understands: 

•  What  the  study  is  about. 

•  What  procedures  will  be  used. 

•  What  the  potential  benefits  might  be. 

•  What  the  known  risks  might  be. 

I  can  confirm  that  this  research  subject  speaks  the  language  that  was  used  to  explain  this  research 
and  is  receiving  an  informed  consent  form  in  the  appropriate  language.  Additionally,  this  subject 
reads  well  enough  to  understand  this  document  or,  if  not,  this  person  is  able  to  hear  and  understand 
when  the  form  is  read  to  him  or  her.  This  subject  does  not  have  a  medical/psychological  problem 
that  would  compromise  comprehension  and  therefore  makes  it  hard  to  understand  what  is  being 
explained  and  can,  therefore,  give  legally  effective  informed  consent.  This  subject  is  not  under  any 
type  of  anesthesia  or  analgesic  that  may  cloud  their  judgment  or  make  it  hard  to  understand  what  is 
being  explained  and,  therefore,  can  be  considered  competent  to  give  informed  consent. 


Signature  of  Person  Obtaining  Informed  Consent  /  Research  Authorization  Date 


Printed  Name  of  Person  Obtaining  Informed  Consent  /  Research  Authorization 
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USF 


UNIVERSITY  OF 
SOUTH  FLORIDA 

Informed  Consent  to  Participate  in  Research  and 
Authorization  to  Collect,  Use  and  Share  Your  Health  Information 

For  Physician 

Information  to  Consider  Before  Taking  Part  in  this  Research  Study 
IRB  Study  #  Pro00000220 


Researchers  at  the  University  of  South  Florida  (USF),  Moffitt  Cancer  Center  and  Tampa  General 
Hospital  (TGH)  study  many  topics.  To  do  this,  we  need  the  help  of  people  who  agree  to  take  part  in  a 
research  study.  This  form  tells  you  about  this  research  study. 

We  are  asking  you  to  take  part  in  a  research  study  that  is  called: 

“Proposal  for  development  of  evidence  based  clinical  decision  support  system  (EBM-CDSS  to  aid 
prognostication  in  terminally  ill  patients” 


The  person  who  is  in  charge  of  this  research  study  is  Dr.  Benjamin  Djulbegovic.  This  person  is  called 
the  Principal  Investigator.  However,  other  research  staff  may  be  involved  and  can  act  on  behalf  of  the 
person  in  charge. 

The  person  explaining  the  research  to  you  may  be  someone  other  than  the  Principal  Investigator. 

Other  research  personnel  who  you  may  be  involved  with  include:  your  consulting  physician  and  or  the 
following  study  personnel:  Catherine  Jahrsdorfer,  Mario  Crawford,  Rahul  Mhaskar  and  Howard  Tuch. 

The  research  will  be  done  at  Tampa  General  Hospital  Moffitt  Cancer  Center  and  Tampa  Bay  Lifepath 
Hospice  and  Palliative  Care. 


This  research  is  being  paid  for  by  the  United  States  Army. 


Purpose  of  the  study 

The  main  purpose  of  this  study  is  to  develop  an  Evidence-based  Clinical  Decision  Support  (EBM- 
CDSS)  system  available  at  bedside  (for  you)  to  improve  prediction  of  the  life  expectancy  of  terminally 
ill  patients  and  improve  referral  of  patients  to  hospice.  The  EBM-CDSS  is  a  tool  that  will  include 
information  using  various  health  related  data  (including  but  not  limited  to  laboratory  values,  disease 
type,  age,  sex,  ethnicity/race,  etc.)  and  also  your  patients’  wishes  and  preferences  made  available  to 
physicians  (like  you)  at  bedside  to  aid  in  better  assessment  of  life  expectancy  of  terminally  ill  patients. 
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We  will  also  request  you  to  share  your  preferences  in  terms  of  continuing  treatment  targeted  at  the 
disease  versus  hospice  care  for  your  patient(s). 

Study  Procedures 

If  you  take  part  in  this  study:  We  will  actively  collect  information  from  you  in  this  study.  The  study 
personnel  will  visit  you  during  the  study  period  and  will  collect  information  from  you.  We  don’t 
expect  each  visit  to  last  more  than  15-20  minutes. 

If  you  take  part  in  this  study,  you  will  be  requested  to  sign  this  informed  consent  form: 

•  You  will  be  requested  to  explain  the  study  in  details  to  your  patients  and  /  or  give  permission  to 
the  study  personnel  to  explain  the  study  in  details  to  your  patients  and  seek  their  participation. 

•  You  will  be  requested  by  the  study  personnel  to  answer  questions  related  to  your  estimates 
regarding  life  expectancy  of  your  patients. 

•  You  will  be  requested  to  answer  questions  regarding  referring  patients  to  hospice  care. 
Specifically,  we  would  like  to  know  your  feelings  of  regret  and  how  you  weigh  the  relative 
harms  of  false-positives  and  false-negatives  regarding  your  decision  of  referring  patient  to 
hospice  care.  For  example:  1)  Imagine  that  you  chose  hospice  care  but  you  should  have 
continued  current  treatment  for  your  patient.  How  much  regret  do  you  think  you  would  feel  in 
this  situation?  2)  Imagine  that  you  chose  to  continue  current  treatment  targeted  at  his/her 
disease  for  your  patient  but  you  should  have  chosen  hospice  care.  How  much  regret  do  you 
think  you  would  feel  in  this  situation?  We  will  also  offer  you  our  best  guess  regarding  the  life 
expectancy  of  a  patient  like  the  one  which  you  are  treating. 

•  We  will  also  offer  you  our  best  advice  regarding  the  choice  of  hospice  care  vs.  continuing 
current  treatment  for  a  patient  like  the  one  which  you  are  treating. 

•  You  will  be  requested  to  answer  questions  to  assess  the  impact  of  our  calculations  (of  life 
expectancy  and  the  recommendation  of  referring  the  patient  to  either  hospice  or  continuing 
current  treatment)  on  your  clinical  judgment. 

•  We  will  also  seek  your  opinion  regarding  usefulness  of  EBM-CDSS  in  treating  your  patients. 

•  We  will  also  offer  our  evidence  based  pain  management  module  to  you.  We  will  also  seek  your 
opinion  regarding  usefulness  of  our  evidence  based  pain  management  module  in  treating  your 
patients. 


Alternatives 

If  you  decide  not  to  take  part  in  this  study,  that  is  okay. 


Benefits 

We  don’t  know  if  you  will  get  any  benefits  by  taking  part  in  this  study.  However,  you  may  find  the 
EBM-CDSS  useful  for  making  decisions  in  care  of  terminally  ill  patients.  That’s  why  we  are  doing  this 
study. 


IRB  Number:  Pro00000220  Version  3  IRB  Consent  Rev.  Date:  Aug  23.  2012 

IC  Adult  Minimal  Risk  RA  Template  -SB  Rev:  9-3-2010  Page  2  of  6 


Risks  or  Discomfort 


This  research  is  considered  to  be  minimal  risk.  That  means  that  the  risks  associated  with  this  study  are 
the  same  as  what  you  face  every  day.  There  are  no  known  additional  risks  to  those  who  take  part  in 
this  study. 

We  understand  that  talking  about  potential  error  in  your  clinical  decision  making  (even  imaginative 
one)  can  be  unpleasant  or  distressing.  If  you  find  anything  what  we  are  about  to  ask  you  upsetting  and 
distressing;  you  may  request  to  discontinue  your  participation  from  the  study.  Your  decision  to 
withhold  from  the  study  will  not  affect  your  status  in  any  way. 


Compensation 

You  will  receive  no  payment  or  other  compensation  for  taking  part  in  this  study. 

Cost 

There  will  be  no  additional  costs  to  you  as  a  result  of  being  in  this  study. 


Conflict  of  Interest  Statement 

There  is  no  conflict  of  interest. 

Confidentiality  of  Information  Used  in  the  Study 

Who  will  see  the  information  that  you  give? 

Not  applicable 

Who  will  disclose  (share),  receive,  and/or  use  your  information? 

Not  applicable 

Who  else  can  use  and  share  this  information? 

Not  applicable 

How  Will  My  Information  Be  Used? 

Not  applicable 

For  the  Research  Participant  (you)  to  complete: 

Not  applicable 

Your  Rights: 
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You  can  refuse  to  sign  this  form.  If  you  do  not  sign  this  form: 

•  You  will  not  be  able  to  take  part  in  this  research. 

•  This  will  not  impact  your  status  outside  of  this  study. 


How  Do  I  Withdraw  Permission  to  Use  My  Information? 

You  can  revoke  this  form  at  any  time  by  sending  a  signed  letter  to  USF  at  the  address  given  below, 
clearly  stating  that  you  wish  to  withdraw  your  consent  to  participate  in  the  research  and  to  the  use  of 
your  information  in  the  research.  If  you  revoke  this  form: 

•  You  will  no  longer  be  a  participant  in  this  research  study. 

•  We  will  stop  collecting  new  information  about  you. 

•  We  will  use  the  information  collected  prior  to  the  revocation  of  your  authorization.  This 
information  may  already  have  been  used  or  shared  with  others,  or  we  may  need  it  to  complete 
and  protect  the  validity  of  the  research.  Staff  may  follow-up  with  you  if  there  is  a  medical 
reason  to  do  so. 

To  revoke  this  form,  you  must  tell  us  in  writing.  Please  write  to: 

Principal  Investigator  Dr.  Benjamin  Djulbegovic 
For  IRB  Study  #  Pro00000220 
12901  Bruce  B.  Downs  Blvd.,  MDC27 
Tampa,  FL  33612 


While  we  are  doing  this  research,  we  cannot  let  you  see  or  copy  the  research  information  we  have 
about  you.  After  the  research  is  done,  you  have  a  right  to  see  the  information  about  you,  as  allowed  by 
USF  policies. 

Privacy  and  Confidentiality 

We  will  keep  your  study  records  private  and  confidential.  Certain  people  may  need  to  see  your  study 
records.  By  law,  anyone  who  looks  at  your  records  must  keep  them  completely  confidential.  The  only 
people  who  will  be  allowed  to  see  these  records  are: 

•  The  research  team,  including  the  Principal  Investigator,  study  coordinator,  research  nurses,  and 
all  other  research  staff. 

•  Certain  government  and  university  people  who  need  to  know  more  about  the  study.  For 
example,  individuals  who  provide  oversight  on  this  study  may  need  to  look  at  your  records. 
This  is  done  to  make  sure  that  we  are  doing  the  study  in  the  right  way.  They  also  need  to  make 
sure  that  we  are  protecting  your  rights  and  your  safety. 

•  Any  agency  of  the  federal,  state,  or  local  government  that  regulates  this  research.  This  includes 
the  Food  and  Drug  Administration  (FDA),  Florida  Department  of  Health,  and  the  Department 
of  Health  and  Human  Services  (DHHS)  and  the  Office  for  Human  Research  Protection 
(OHRP). 

•  The  USF  Institutional  Review  Board  (IRB)  and  its  related  staff  who  have  oversight 
responsibilities  for  this  study,  staff  in  the  USF  Office  of  Research  and  Innovation,  USF 


IRB  Number:  Pro00000220  Version  3  IRB  Consent  Rev.  Date:  Aug  23.  2012 

IC  Adult  Minimal  Risk  RA  Template  -SB  Rev:  9-3-2010  Page  4  of  6 


Division  of  Research  Integrity  and  Compliance,  and  other  USF  offices  who  oversee  this 
research. 

•  The  sponsors  of  this  study  The  United  States  Army 

We  may  publish  what  we  learn  from  this  study.  If  we  do,  we  will  not  include  your  name.  We  will  not 
publish  anything  that  would  let  people  know  who  you  are. 


Voluntary  Participation  /  Withdrawal 

You  should  only  take  part  in  this  study  if  you  want  to  volunteer.  You  should  not  feel  that  there  is  any 
pressure  to  take  part  in  the  study,  to  please  the  investigator  or  the  research  staff.  You  are  free  to 
participate  in  this  research  or  withdraw  at  any  time.  There  will  be  no  penalty  or  loss  of  benefits  you 
are  entitled  to  receive  if  you  stop  taking  part  in  this  study. 

New  information  about  the  study 

During  the  course  of  this  study,  we  may  find  more  information  that  could  be  important  to  you.  This 
includes  information  that,  once  learned,  might  cause  you  to  change  your  mind  about  being  in  the  study. 
We  will  notify  you  as  soon  as  possible  if  such  information  becomes  available. 


You  can  get  answers  to  your  questions,  concerns,  or  complaints 

If  you  have  any  questions,  concerns  or  complaints  about  this  study,  call  Dr.  Benjamin  Djulbegovic  at 
813-396-9178. 

If  you  have  questions  about  your  rights  as  a  participant  in  this  study,  general  questions,  or  have 
complaints,  concerns  or  issues  you  want  to  discuss  with  someone  outside  the  research,  call  the 
Division  of  Research  Integrity  and  Compliance  of  the  University  of  South  Florida  at  (813)  974-5638. 

Consent  to  Take  Part  in  this  Research  Study 

It  is  up  to  you  to  decide  whether  you  want  to  take  part  in  this  study.  If  you  want  to  take  part,  please 
sign  the  form,  if  the  following  statements  are  true. 

I  freely  give  my  consent  to  take  part  in  this  study.  I  understand  that  by  signing  this  form  I  am 
agreeing  to  take  part  in  research.  I  have  received  a  copy  of  this  form  to  take  with  me. 


Signature  of  Person  Taking  Part  in  Study 


Date 


Printed  Name  of  Person  Taking  Part  in  Study 


Statement  of  Person  Obtaining  Informed  Consent  and  Research  Authorization 

I  have  carefully  explained  to  the  person  taking  part  in  the  study  what  he  or  she  can  expect. 
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I  hereby  certify  that  when  this  person  signs  this  form,  to  the  best  of  my  knowledge,  he  or  she 
understands: 

•  What  the  study  is  about. 

•  What  procedures  will  be  used. 

•  What  the  potential  benefits  might  be. 

•  What  the  known  risks  might  be. 

•  I  can  confirm  that  this  research  subject  speaks  the  language  that  was  used  to  explain  this 
research  and  is  receiving  an  informed  consent  form  in  the  appropriate  language.  Additionally, 
this  subject  reads  well  enough  to  understand  this  document  or,  if  not,  this  person  is  able  to  hear 
and  understand  when  the  form  is  read  to  him  or  her.  This  subject  does  not  have  a 
medical/psychological  problem  that  would  compromise  comprehension  and  therefore  makes  it 
hard  to  understand  what  is  being  explained  and  can,  therefore,  give  legally  effective  informed 
consent.  This  subject  is  not  under  any  type  of  anesthesia  or  analgesic  that  may  cloud  their 
judgment  or  make  it  hard  to  understand  what  is  being  explained  and,  therefore,  can  be 
considered  competent  to  give  informed  consent. 


Signature  of  Person  Obtaining  Informed  Consent  /  Research  Authorization  Date 


Printed  Name  of  Person  Obtaining  Informed  Consent  /  Research  Authorization 
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USF 


UNIVERSITY  OF 
SOUTH  FLORIDA 


Informed  Consent  of  Legally  Authorized  Representative 
(Proxy/Healthcare  Surrogate)  for  Participation  in  Research 

Information  to  Consider  Before  Taking  Part  in  This  Research  Study 
IRB  Study  #  Pro00000220 


Researchers  at  the  Tampa  General  Hospital,  Moffitt  Cancer  Center  and  University  of  South  Florida 
study  many  topics.  Our  goal  is  to  find  better  ways  to  help  treat  patients.  To  do  this,  we  need  the  help  of 
people  who  agree  to  take  part  in  a  research  study. 

We  are  asking  you  to  take  part  in  a  research  study  that  is  called: 

“Proposal  for  development  of  evidence  based  clinical  decision  support  system  (EBM-CDSS)  to  aid 
prognostication  in  terminally  ill  patients” 

We  are  approaching  you  to  ask  you  to  participate  in  this  study  because  we  were  told  that  you  have  a 
terminal  disease  that  may  significantly  shorten  your  life  duration.  You  may  participate  in  this  study; 
only  if  you  aware  that  you  have  you  have  a  terminal  disease. 

The  person  who  is  in  charge  of  this  research  study  is  Dr.  Benjamin  Djulbegovic.  This  person  is  called 
the  Principal  Investigator.  However,  other  research  staff  may  be  involved  and  can  act  on  behalf  of  the 
person  in  charge.  The  person  explaining  the  research  to  you  may  be  someone  other  than  the  Principal 
Investigator. 

Other  research  personnel  who  may  be  involved  with  you  include:  your  consulting  physician  and  or  the 
following  study  personnel:  Catherine  Jahrsdorfer,  Mario  Crawford  ,  Rahul  Mhaskar  and  Howard  Tuch 

The  research  will  be  done  at  Tampa  General  Hospital  Tampa  Bay  Lifepath  Hospice  and  Palliative  Care 
and  Moffitt  cancer  center. 

This  research  is  being  paid  for  by  the  United  States  Army. 


Finding  the  best  person  to  give  consent  by  proxy 

Under  certain  circumstances,  someone  can  give  consent  and  research  authorization  for  another  person 
to  take  part  in  this  research  study.  This  person  is  the  “subject  by  proxy.”  The  proxy  can  make  choices 
for  the  subject,  if  the  subject  is  not  able  to  make  choices  for  him  or  herself.  This  person  serves  as  the 
legally  authorized  representative.  A  proxy  can  be  any  of  the  people  listed  in  A. 

A.  Look  at  the  list  and  write  Proxy  in  the  space  next  to  the  description  of  the  person  who  will  give 
consent  and  research  authorization  for  the  person  participating  in  this  research.  If,  there  is  a  person 
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with  a  higher  authority,  write  in  the  space  why  that  person  is  not  available,  willing,  or  able  to  act  as 
proxy.  This  following  is  an  example: 

Health  Care  Surrogate:  No  one  was  named 
Spouse:  Spouse  has  died 

Adult  Child:  Unable  to  reach  by  phone  after  several  tries 

Parent:  PROXY 

(1)  Health  Care  Surrogate  named  by  person: _ 

(2)  A  guardian  of  the  person,  appointed  by  the  court.  He/she  must  be  authorized  to  give  consent 

to  medical  treatment: _ 

(3)  The  person's  spouse: _ 

(4)  An  adult  child  of  the  person.  If  the  person  has  more  than  one  adult  child,  a  majority  of 

the  adult  children  who  live  near  enough  to  be  asked: _ 

(5)  A  parent  of  the  person: _ 

(6)  The  adult  sibling  of  the  person.  If  the  person  has  more  than  one  sibling,  a  majority  of  the  adult 
siblings  who  live  near  enough  to  know  what  is  going  on: 


(7)  An  adult  relative  of  the  person  who  has  shown  special  care  and  concern  for  the  person. 

This  adult  relative  has  kept  regular  contact  with  the  person.  He/she  knows  how  the  person  feels 
about  things,  what  the  person  likes  to  do,  what  the  person’s  health  is  like,  what  the  person 
believes  and  thinks  is  right 


(8)  A  close  friend  of  the  person: _ . 

(9)  A  clinical  social  worker  licensed  pursuant  to  Chapter  491,  or  who  is  a  graduate  of  a  court- 
approved  guardianship  program.  Such  a  proxy  must  be  selected  by  the  provider’s  bioethics 
committee  and  must  not  be  employed  by  the  provider.  If  the  provider  does  not  have  a  bioethics 
committee,  then  such  a  proxy  may  be  chosen  through  an  arrangement  with  the  bioethics 
committee  of  another  provider.  The  proxy  will  be  notified  that,  upon  request,  the  provider  shall 
make  available  a  second  physician,  not  involving  in  the  patient’s  care  to  assist  the  proxy  in 
evaluating  treatment.  Decisions  to  withhold  or  withdraw  life-prolonging  procedures  will  be 
reviewed  by  the  facility’s  bioethics  committee.  Documentation  of  efforts  to  locate  proxies 
from  prior  classes  must  be  recorded  in  the  patient  record. 

Proxy’s  Statement  of  Consent: 

I  understand  that  I  am  being  asked  to  serve  as  the  legally  authorized  representative  for  (name  of 

participant) _ and  give  permission  for  him/her  to 

participate  in  this  research  study.  My  decision  is  based  on  what  I  believe  the  person  would  choose  for 
him/herself  and  what  I  believe  is  now  best  for  the  person,  based  on  the  information  I  have  been 
provided. 

Should  the  person  for  whom  you  are  signing  consent  take  part  in  this  study? 

This  form  tells  you  about  this  research  study.  After  reading  through  this  form  and  having  the  research 
explained  to  you  by  someone  conducting  this  research,  you  can  decide  if  you  think  the  person  for 
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whom  you  are  signing  consent.  Reading  this  form  should  help  you  decide  whether  the  person  for 
whom  you  are  signing  consent  would  want  to  take  part  in  the  study.  If,  at  any  time,  you  have  any 
questions,  feel  free  to  ask  the  person  explaining  this  study  to  you. 

The  remainder  of  this  form  is  written  as  if  you,  the  Legally  Authorized  Representative,  were 
participating  in  the  research.  This  helps  you  think  in  terms  of  what  the  person  for  whom  you  are 
signing  consent  would  do  or  what  is  best  for  that  person. 

Before  you  decide: 

•  Read  this  form. 

.  Have  a  friend  or  family  member  read  it. 

.  Talk  about  this  study  with  the  person  in  charge  of  the  study  or  the  person  explaining  the  study. 
You  can  have  someone  with  you  when  you  talk  about  the  study. 

•  Talk  it  over  with  someone  you  trust. 

•  Find  out  what  the  study  is  about. 

.  You  may  have  questions  this  form  does  not  answer.  You  do  not  have  to  guess  at  things  you 
don’t  understand.  If  you  have  questions  ask  the  person  in  charge  of  the  study  or  study  staff  as 
you  go  along.  Ask  them  to  explain  things  in  a  way  you  can  understand. 

•  Take  your  time  to  think  about  it. 

It  is  up  to  you.  If  you  choose  to  take  part  in  this  study,  you  will  need  to  sign  this  consent  form.  If 

you  do  not  want  to  take  part  in  this  study,  you  should  not  sign  this  form. 

Why  is  this  research  being  done? 

The  purpose  of  this  study  is  to  help  people  in  the  final  stages  of  their  lives  make  decisions  about 
continuing  medical  treatment  targeting  their  disease  or  choosing  hospice  care.  Hospice  is  a  type  of  care 
that  provides  support  to  people  at  the  end  of  their  lives.  The  main  goal  of  hospice  care  is  to  improve 
the  quality  of  patients’  lives  by  focusing  on  pain  and  other  symptom  relief  and  emotional  comfort.  This 
means  that  medical  treatments  that  are  given  to  target  your  disease  may  be  discontinued. 

This  research  study  is  aimed  at  helping  you  choose  what  type  of  medical  care  you  want  to  have  during 
the  upcoming  months.  Most  people  in  this  situation  are  uncertain  how  they  want  to  approach  their 
illness.  We  have  created  a  tool  to  help  people  and  their  doctors  make  the  choice  that  they  feel  is  right 
for  them.  The  tool  is  designed  to  help  doctors  and  patients  like  yourself  decide  whether  to  continue 
treatment  targeting  their  disease  although  the  treatment  may  create  additional  discomfort,  or  whether  to 
choose  care  through  hospice  which  is  focused  on  providing  comfort  and  symptom  relief.  Either  of 
these  choices  may  or  may  not  prolong  your  life.  We  would  also  like  to  understand  your  wishes  and 
preferences  to  help  you  make  the  choice  that  is  right  for  you. 

The  tool  that  is  being  tested  in  this  research  is  called  an  Evidence-based  Medicine  Clinical  Decision 
Support  System  (EBM-CDSS).  The  EBM-CDSS  is  designed  to  be  used  by  doctors  at  the  bedside  to 
better  predict  life  expectancy  in  patients  with  terminal  illness  and  to  improve  the  timing  and 
appropriateness  of  referral  to  hospice  care. 
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We  are  asking  you  to  take  part  in  this  research  because  you  have  an  illness  that 
may  significantly  shorten  your  life  span.  Why  are  you  being  asked  to  take  part? 

We  are  asking  you  to  take  part  in  this  research  study  because  you  are  one  of  the  patients  being 
referred  to  hospice  care. 

What  will  happen  during  this  study? 

•  If  you  take  part  in  this  study:  We  will  actively  collect  information  from  you  until  your  demise. 
The  study  personnel  or  your  attending  physician  will  visit  you  twice  initially  and  then  once  a 
month  during  the  study  period  and  will  collect  information  from  you.  We  do  not  expect  the 
initial  visits  to  last  more  than  30  minutes  and  the  monthly  visits  to  last  more  than  10  minutes. 

•  If  you  take  part  in  this  study,  you  will  be  requested  to  sign  this  informed  consent  form: 

•  After  signing  the  informed  consent  you  will  be  requested  by  the  study  personnel  to  share 
certain  information  related  to  your  health  from  your  medical  records.  You  may  be  requested  to 
answer  questions  regarding  your  health. 

•  The  interviewer  will  also  ask  you  whether  you  would  like  to  know  your  doctor’s  best  estimate 
of  your  life  expectancy. 

•  We  will  estimate  your  life  expectancy  based  on  our  current  best  estimates  for  “average” 
patients  like  you  (i.e.  estimate  of  the  average  future  lifetime  for  people  like  you)?  We  will  ask 
you  whether  you  would  like  to  know  the  best  estimates  of  your  life  expectancy  with  the  current 
treatment  targeted  at  your  disease  and  without  this  treatment. 

•  To  help  you  decide  between  hospice  care  and  treatment  targeted  at  your  disease.  We  will  also 
like  to  learn  about  your  wishes  and  preferences  regarding  how  you  would  like  to  approach  your 
illness.  We  will  do  that  by  using  our  tool  to  understand  your  feelings  of  regret  in  case  you  made 
a  wrong  choice. 

•  The  interviewer  will  also  seek  your  opinion  regarding  your  tolerance  if  your  doctor  is  mistaken 
regarding  whether  to  continue  your  present  treatment  (that  may  potentially  cure  your  disease  or 
prolong  your  life)  or  to  discontinue  your  treatment  and  refer  you  to  hospice  care.  We  will  ask 
you  a  series  of  questions  assuming  that  there  is  an  error  in  prediction  of  your  life  expectancy 
with  and  without  treatment  targeted  at  your  disease.  We  want  to  know  how  large  an  error  you 
can  tolerate  and  still  not  regret  (magnitude  of  acceptable  regret  for  you)  your  doctor’s  decision 
to  be  referred  to  hospice  care.  After  you  make  the  decision  regarding  choosing  hospice  care  or 
continuing  treatment  we  will  ask  you  questions  to  obtain  your  opinion  regarding  the  decision 
you  made  once  a  month  until  you  pass  away. 

•  We  will  ask  you  whether  you  changed  your  decision  for  enrolling  /  not  enrolling  in  hospice 
care  based  on  the  information  provided  to  you  by  us. 

•  You  should  know  that  we  cannot  tell  you  which  of  these  choices  (treatment  targeted  at  your 
disease  versus  hospice  care)  will  prolong  your  life.  However,  we  may  be  able  to  tell  you  which 
of  these  choices  are  most  compatible  with  your  wishes  and  preferences  for  an  average  patient 
like  you. 

How  many  other  people  will  take  part? 

About  400  people  will  take  part  in  this  study. 
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What  other  choices  do  you  have  if  you  decide  not  to  take  part? 

You  do  not  have  to  take  part  in  this  study.  Your  decision  to  take  part  in  this  study  or  not  to  take  part 
will  not  affect  the  medical  care  that  you  receive  You  can  choose  to  fully  take  part  in  this  study 
including  the  collection  of  your  medical  record  information  and  the  initial  and  follow  up  interviews. 
You  can  also  choose  to  partially  participate  in  this  study  either  by: 

•  allowing  us  to  look  at  your  medical  records  only  and  NOT  take  part  in  the  interview  process 
OR 

•  NOT  allow  us  to  look  at  your  medical  records  or  interview  you  but  ONLY  allow  us  to  collect 
the  date  that  you  eventually  pass  away. 

You  choice  of  whether  to  take  part  in  the  study  fully  or  one  of  the  partial  options  will  be  indicated 
on  the  last  pages  of  this  consent  form.  There  is  a  separate  signature  area  for  each  choice  and  only 
one  of  these  sections  should  be  signed. 


Will  you  be  paid  for  taking  part  in  this  study? 

You  will  receive  no  payment  or  other  compensation  for  taking  part  in  this  study. 
What  will  it  cost  you  to  take  part  in  this  study? 

There  will  be  no  additional  costs  to  you  as  a  result  of  being  in  this  study.  However,  routine  medical 
care  for  your  condition  (care  you  would  have  received  whether  or  not  you  were  in  this  study)  will  be 
charged  to  you  or  your  insurance  company. 


What  are  the  potential  benefits  if  you  take  part  in  this  study? 

We  don’t  know  if  you  will  get  any  benefits  by  taking  part  in  this  study.  However,  this  study  may 
improve  doctor’s  accuracy  to  assess  life  expectancy.  Many  people  find  this  important  which  can  be 
important  for  you  too.  Many  people  also  find  it  helpful  to  clarify  their  wishes  and  preferences 
regarding  how  they  would  like  to  spend  the  reminder  of  their  life. 


What  are  the  risks  if  you  take  part  in  this  study? 

There  are  minimal  risks  for  participating  in  this  study.  While  many  people  in  your  situations  want  to 
have  a  frank  conversation  about  doctors’  best  estimate  of  their  life  expectancy,  we  understand  that 
talking  about  how  to  approach  your  illness  is  difficult  and  can  be  very  distressing.  If  you  find  anything 
that  we  are  about  to  ask  you  upsetting  and  distressing,  you  may  request  to  discontinue  your 
participation  from  the  study.  There  is  no  funding  to  compensate  you  for  treatment  if  “injury”  occurs. 
There  might  be  other  unforeseeable  risks  that  could  occur  which  are  unknown  at  this  time.  Your 
decision  to  withdraw  from  the  study  will  not  affect  the  care  you  will  receive  in  any  form  and  or 
manner.  If  you  wish  to  consult  the  study  Principal  Investigator  and  Co-  Principal  Investigator  (they 
have  extensive  experience  working  with  patients  in  similar  situations)  will  immediately  be  available 
for  counseling. 
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Conflict  of  Interest  Statement 

There  is  no  conflict  of  interest. 


Authorization  to  Use  and  Disclose  Protected  Health  Information 

Who  will  see  your  health  information? 

In  our  research,  we  use  and  share  your  health  information  to  the  extent  authorized  by  you.  We  know 
that  this  information  is  private.  The  federal  privacy  regulations  of  the  Health  Insurance  Portability  & 
Accountability  Act  (HIPAA)  protect  your  identifiable  health  information.  If  you  authorize  us  to  use 
your  information  we  will  protect  it  as  required  by  the  law. 

Research  at  Tampa  General  Hospital  and  Moffitt  cancer  center  is  conducted  jointly  with  the  University 
of  South  Florida.  By  signing  this  form,  you  are  permitting  Tampa  General  Hospital, Moffitt  cancer 
center  and  University  of  South  Florida  to  use  personal  health  information  collected  about  you  for 
research  purposes.  You  are  also  allowing  Tampa  General  Hospital  and  Moffitt  cancer  center  to  share 
your  personal  health  information  with  individuals  or  organizations  other  than  USF,  Moffitt  cancer 
center  and  Tampa  General  Hospital  who  are  also  involved  in  this  research  and  listed  below. 


Who  will  disclose  (share),  receive,  and/or  use  your  information? 

To  do  this  research,  USF  and  the  people  and  organizations  listed  below  may  use  or  share  your 
information.  They  may  only  use  and  share  your  information: 

•  With  the  people  and  organizations  on  this  list; 

•  With  you  or  your  personal  representative;  and 

•  As  allowed  by  law. 

In  addition  to  the  people  and  organizations  listed  below  in  the  Privacy  and  Confidentiality  section  of 
this  document,  the  following  groups  of  people  may  also  be  able  to  see  information  about  you  and  may 
use  the  information  to  conduct  the  research: 

•  The  medical  staff  that  takes  care  of  you  and  those  who  are  part  of  this  research  study; 

•  Each  research  site  for  this  study.  This  includes  the  research  and  medical  staff  at  each  site  and 
USF; 

•  The  designated  peer  review  committees  such  as  the  Tampa  General  Hospital  and  Moffitt  cancer 
center  Feasibility  Committee; 

•  Additionally,  there  may  be  other  people  and/or  organizations  that  may  be  given  access  to  your 
personal  health  information.  This  includes  Tampa  Bay  LifePath  Hospice  care. 


Who  else  can  use  and  share  this  information? 

Anyone  listed  above  may  use  consultants  in  this  research,  and  for  the  purpose  of  this  study  may  share 
your  information  with  them.  If  you  have  questions  about  who  they  are,  you  can  ask  us.  Individuals 
who  receive  your  health  information  for  this  research  study  may  not  be  required  by  the  HIPAA  Privacy 
Rule  to  protect  it  and  may  share  your  information  with  others  without  your  permission.  They  can  do  so 
if  permitted  by  the  laws  governing  them.  Example:  The  sponsor  may  share  your  information.  If  the 
sponsor  or  others  share  your  information,  your  information  may  no  longer  be  protected  under  the 
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HIP  A  A  Privacy  Rule. 

How  Will  My  Information  Be  Used? 

By  signing  this  form,  you  are  giving  your  permission  to  use  and/or  share  your  health  information  as 
described  in  this  document  for  any  and  all  study/research  related  purposes.  For  example,  your 
information  may  be  used  as  necessary  for  your  research-related  treatment,  to  collect  payment  for  your 
research-related  treatment  (when  applicable)  and  to  conduct  regular  business  operations.  Your 
authorization  (permission)  to  use  your  health  information  will  not  expire  until  the  end  of  this  research 
study  unless  you  revoke  this  authorization  in  writing. 

As  part  of  the  research,  USF  may  collect,  use,  and  share  the  following  information: 

•  Your  whole  research  record 

•  All  of  your  past,  current  or  future  medical  and  other  health  records  held  by  USF,  other  health 
care  providers  or  any  other  site  affiliated  with  this  study.  This  includes,  but  is  not  limited  to, 
HIV/AIDs,  mental  health,  substance  abuse,  and/or  genetic  information 

We  may  publish  what  we  learn  from  this  study.  If  we  do,  we  will  not  let  anyone  know  your  name.  We 
will  not  publish  anything  else  that  would  let  people  know  who  you  are. 

Your  Rights: 

You  can  refuse  to  sign  this  form.  If  you  do  not  sign  this  form: 

•  You  will  not  be  able  to  take  part  in  this  research  and  therefore  not  be  able  to  receive  the 
research  related  intervention. 

•  This  will  not  change  your  health  care  outside  of  this  study. 

•  This  will  not  change  your  health  care  benefits. 

•  This  will  not  change  the  costs  of  your  health  care. 


How  Do  I  Withdraw  Permission  to  Use  My  Information? 

You  can  revoke  this  form  at  any  time  by  sending  a  signed  letter  to  USF  at  the  address  given  below, 
clearly  stating  that  you  wish  to  withdraw  your  authorization  to  the  use  of  your  health  information  in  the 
research.  If  you  revoke  you  permission: 

•  You  will  no  longer  be  a  participant  in  this  research  study. 

•  We  will  stop  collecting  new  information  about  you. 

•  We  will  use  the  information  collected  prior  to  the  revocation  of  your  authorization.  This 
information  may  already  have  been  used  or  shared  with  others,  or  we  may  need  it  to  complete 
and  protect  the  validity  of  the  research. 

•  Staff  may  follow-up  with  you  if  there  is  a  medical  reason  to  do  so. 

To  revoke  this  form,  you  must  tell  us  in  writing.  Please  write  to: 

Principal  Investigator  Benjamin  Djulbegovic 
For  IRB  Study  #  [Pro00000220 
12901  Bruce  B.  Downs  Blvd.,  MDC27 
Tampa,  FL  33612 
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Privacy  and  Confidentiality 

We  will  keep  your  study  records  private  and  confidential.  Certain  people  may  need  to  see  your  study 
records.  By  law,  anyone  who  looks  at  your  records  must  keep  them  completely  confidential.  The  only 
people  who  will  be  allowed  to  see  these  records  are: 

•  The  research  team,  including  the  Principal  Investigator,  study  coordinator,  research  nurses,  and 
all  other  research  staff. 

•  Certain  government  and  university  people  who  need  to  know  more  about  the  study.  For 
example,  individuals  who  provide  oversight  on  this  study  may  need  to  look  at  your  records. 
This  is  done  to  make  sure  that  we  are  doing  the  study  in  the  right  way.  They  also  need  to  make 
sure  that  we  are  protecting  your  rights  and  your  safety. 

•  Any  agency  of  the  federal,  state,  or  local  government  that  regulates  this  research.  This  includes 
the  Food  and  Drug  Administration  (FDA),  Florida  Department  of  Health,  and  the  Department 
of  Health  and  Human  Services  (DHHS)  and  the  Office  for  Human  Research  Protection 
(OHRP). 

•  The  USF  Institutional  Review  Board  (IRB)  and  its  related  staff  who  have  oversight 
responsibilities  for  this  study,  staff  in  the  USF  Office  of  Research  and  Innovation,  USF 
Division  of  Research  Integrity  and  Compliance,  and  other  USF  offices  who  oversee  this 
research. 

•  The  sponsors  of  this  study  The  United  States  Army 

We  may  publish  what  we  learn  from  this  study.  If  we  do,  we  will  not  include  your  name.  We 
will  not  publish  anything  that  would  let  people  know  who  you  are. 


What  happens  if  you  decide  not  to  take  part  in  this  study? 

You  should  only  take  part  in  this  study  if  you  want  to  volunteer.  You  should  not  feel  that  there  is  any 
pressure  to  take  part  in  the  study  to  please  the  study  doctor  or  the  research  staff. 

If  you  decide  not  to  take  part: 

•  You  will  not  be  in  trouble  or  lose  any  rights  you  normally  have. 

•  You  will  still  have  the  same  services  you  would  normally  have. 

•  You  can  still  get  your  regular  therapy/counseling/services  from  your  regular 
therapist/counselor/social  worker 

You  can  decide  after  signing  this  informed  consent  document  that  you  no  longer  want  to  take  part  in 
this  study.  We  will  keep  you  informed  of  any  new  developments  which  might  affect  your 
willingness  to  continue  to  participate  in  the  study.  However,  you  can  decide  you  want  to  stop 
taking  part  in  the  study  for  any  reason  at  any  time.  If  you  decide  you  want  to  stop  taking  part  in 
the  study,  tell  the  study  staff  as  soon  as  you  can. 

•  We  will  tell  you  how  to  stop  safely.  We  will  tell  you  if  there  are  any  dangers  if  you  stop 
suddenly. 

•  If  you  decide  to  stop,  you  can  go  on  getting  your  regular  care  from  your  physician. 
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Even  if  you  want  to  stay  in  the  study,  there  may  be  reasons  we  will  need  to  take  you  out  of  it. 

You  may  be  taken  out  of  this  study  if: 

•  We  find  out  it  is  not  safe  for  you  to  stay  in  the  study.  For  example,  your  health  may  get  worse. 

•  You  are  not  coming  for  your  study  visits  when  scheduled. 

You  can  get  the  answers  to  your  questions,  concerns,  or  complaints. 

If  you  have  any  questions,  concerns  or  complaints  about  this  study  or  experience  an  adverse  event  or 
unanticipated  problem,  call  Dr.  Benjamin  Djulbegovic  at  813-396-9178. 

If  you  have  questions  about  your  rights,  general  questions,  complaints,  or  issues  as  a  person  taking  part 
in  this  study,  call  the  Division  of  Research  Integrity  and  Compliance  of  the  University  of  South  Florida 
at  (813)  974-5638. 


New  information  about  the  study 

During  the  course  of  this  study,  we  may  find  more  information  that  could  be  important  to  you.  This 
includes  information  that,  once  learned,  might  cause  you  to  change  your  mind  about  being  in  the  study. 
We  will  notify  you  as  soon  as  possible  if  such  information  becomes  available. 


Signature  of  Legally  Authorized  Representative  (Proxy  /  Health  Care  Surrogate) 

Full  Consent  to  Take  Part  in  this  Research,  including  medical  record  review,  initial  interview 
and  follow  up  interviews: 


It  is  up  to  you  to  decide  whether  you  want _ 

(name  of  participant)  to  take  part  in  this  study.  If  you  want  this  person  to  take  part,  please  read  the 
statements  below  and  sign  the  form  if  the  statements  are  true. 

I  freely  give  my  consent  to  have _ (name 

of  participant)  take  part  in  this  study.  I  understand  that  I  am  allowing  the  research  team  to  access  the 
patient’s  health  records  for  the  purpose  of  this  study  and  interview  me  regarding  the  patient’s  health 
and  his/hers  decision  regarding  enrolling  for  hospice  care.  I  understand  that  by  signing  this  form  I  am 
agreeing  for  the  individual  named  above  to  take  part  in  research.  I  have  received  a  copy  of  this  form  to 
take  with  me. 


Signature  of  Legally  Authorized  Representative 


Date 


Printed  Name  Legally  Authorized  Representative 
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Partial  consent  to  take  part  in  the  research,  without  participation  in  interview 

process 

It  is  up  to  you  to  decide  whether  you  want _ 

(name  of  participant)  to  take  part  in  this  study.  If  you  want  this  person  to  take  part,  please  read  the 
statements  below  and  sign  the  form  if  the  statements  are  true. 

I  freely  give  my  consent  to  have _ (name 

of  participant)  partially  take  part  in  this  study.  I  understand  that  I  am  allowing  the  research  team  to 
access  the  patient’s  health  records  for  the  purpose  of  this  study.  I  understand  that  by  signing  this  form 
I  am  agreeing  for  the  individual  named  above  to  take  part  in  research.  I  have  received  a  copy  of  this 
form  to  take  with  me. 

I  am  allowing  the  researchers  to  access  my  medical  record  and  I  am  NOT  willing  to  participate  in  the 
interview  process. 


Signature  of  Legally  Authorized  Representative 


Date 


Printed  Name  Legally  Authorized  Representative 

Partial  consent  to  take  part  in  the  research,  Without  Participation  in  interview 

process  or  medical  record  review 


It  is  up  to  you  to  decide  whether  you  want _ 

(name  of  participant)  to  take  part  in  this  study.  If  you  want  this  person  to  take  part,  please  read  the 
statements  below  and  sign  the  form  if  the  statements  are  true. 

I  freely  give  my  consent  to  have _ (name 

of  participant)  partially  take  part  in  this  study.  I  understand  that  I  am  allowing  the  research  team  to 
access  only  the  date  of  the  individual’s  named  above  date  of  demise  for  the  purpose  of  this  study.  I 
understand  that  by  signing  this  form  I  am  agreeing  for  the  individual  named  above  to  take  part  in 
research.  I  have  received  a  copy  of  this  form  to  take  with  me. 

I  understand  that  I  am  allowing  the  research  team  to  collect  the  date  that  I  eventually  pass  away  for 
the  purpose  of  this  study.  I  understand  that  by  signing  this  form  I  am  agreeing  to  take  part  in  research. 

I  have  received  a  copy  of  this  form  to  take  with  me. 

I  ONLY  ALLOW  researchers  to  access  medical  records  to  collect  the  date  that  I  eventually  pass  away. 


Signature  of  Legally  Authorized  Representative 


Date 


Printed  Name  Legally  Authorized  Representative 
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Determination  of  the  Person’s  Ability  to  Give  Consent 

A.  I  am  (participant’s  name) _ ’s _ (relationship). 

I  have  examined  this  individual  and  have  found  that  he/she  is  unable  to  give  informed  consent  to 
take  part  in  the  research  study. 

Signature  of  Person  Attesting  to  Limited/Diminished  Autonomy  of  Participant  Date 
Printed  Name  of  Person  Attesting  to  Limited/Diminished  Autonomy  of  Participant 

-OR- 

B.  I  am  a  physician  licensed  in  the  State  of  Florida.  I  agree  that  this  person  is  unable  to  give  consent. 


Signature  of  Physician 


Date 


Printed  Name  of  Physician 

Paternal  Consent 

If  you  were  pregnant  or  become  pregnant,  the  particular  treatment  or  procedure  might  involve  risks  to 
the  embryo  or  fetus,  which  were  currently  unforeseeable. 


Signature  of  father  of  unborn  child  Date 

The  signature  of  the  father  was  not  obtained  because: 

□  He  was  not  readily  available 

□  Other  reasons; _ 

Statement  of  Person  Obtaining  Informed  Consent 

I  have  carefully  explained  to  the  person  who  is  giving  consent  for _ 

who  is  taking  part  in  the  study  what  he  or  she  can  expect. 

I  hereby  certify  that  when  this  person  signs  this  form,  to  the  best  of  my  knowledge,  he  or  she 
understands: 

•  What  the  study  is  about. 

•  What  procedures/interventions/investigational  drugs  or  devices  will  be  used. 

•  What  the  potential  benefits  might  be. 

•  What  the  known  risks  might  be. 

•  How  the  information  collected  about  the  person  will  be  used. 

I  also  certify  that  he  or  she  does  not  have  any  problems  that  could  make  it  hard  to  understand  what  it 
means  to  take  part  in  this  research.  This  person  speaks  the  language  that  was  used  to  explain  this 
research.  This  person  reads  well  enough  to  understand  this  form  or,  if  not,  this  person  is  able  to  hear 
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and  understand  when  the  form  is  read  to  him  or  her. 

This  person  does  not  have  a  medical/psychological  problem  that  would  compromise  comprehension 
and  therefore  makes  it  hard  to  understand  what  is  being  explained  and  can,  therefore,  give  informed 
consent.  This  person  is  not  under  any  type  of  anesthesia  or  analgesic  that  may  cloud  their  judgment  or 
make  it  hard  to  understand  what  is  being  explained  and,  therefore,  can  be  considered  competent  to  give 
informed  consent. 


Signature  of  Person  Obtaining  Informed  Consent 


Date 


Printed  Name  of  Person  Obtaining  Informed  Consent 
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Appendix  11  Prospective  phase  interview  guidelines  and  software  user  manual 


Interview  guide 

Please  use  the  following  interview  guide  for  data  collection  from  participating  physicians  and 
patients. 

You  will  be  using  the  Hospice  Referral  software  in  addition  to  the  case  report  forms  (hard 
copies:  appendix  1)  to  record  data  obtained  from  participants.  You  will  be  provided  with  a 
laptop  computer  in  which  the  Hospice  Referral  Data  Collection  Software  (Hospice  Referral 
software)  will  be  pre-installed.  The  software  will  perform  all  computations  required  to  derive 
the  best  advice  for  each  decision  maker  (e.g.  healthcare  provider,  patient  or  surrogate) 
according  to  the  prognostication  models  and  the  decision  maker's  preferences. 

Hospice  Referral  Data  Collection  Software  -  User  guide 

To  run  the  Hospice  Referral  software,  you  need  to  locate  it  in  the  computer's  desktop  and  then 
double  click  the  shortcut  associated  with  the  software.  The  shortcut  will  be  named:  Hospice 
Referral.  This  shortcut  will  be  depicted  in  the  computer's  desktop  with  the  following  icon: 


Hospice 

Referral 


USF EBM 


Once  the  software  has  started  you  will  see  the  following  welcome  screen: 


Click  on  "1st  interview"  button  only  when  you  interview  a  patient/surrogate  for  the  first  time  or 
when  you  interview  a  healthcare  provider  regarding  a  specific  patient  for  the  first  time  (regret 
interview).  Otherwise  click  on  the  "Follow  up"  button  (post  decisonal  regret  interview).  The 
button  "Exit"  will  terminate  the  software.  The  button  "1st  interview"  is  intended  for 
patient/surrogate  and  healthcare  provider  interviews.  The  button  "Follow  up"  is  intended  only 
for  patient/surrogate  interviews. 

The  "1st  interview"  button  will  lead  you  to  the  following  screen: 


The  software  uses  tabs  to  present  the  information  in  an  organized  manner.  To  avoid  confusion 
of  switching  between  tabs,  all  tabs  are  invisible  until  needed.  You  can  navigate  FORWARD 
between  tabs  of  the  Hospice  Referral  software  by  clicking  the  NEXT  button  on  the  bottom  of 
the  screen.  Once  a  tab  is  activated,  it  will  remain  this  way  until  a  new  interview  is  initiated.  You 
can  navigate  BACKWARDS  between  tabs  by  clicking  on  the  name  tag  of  the  desired  tab. 

The  CANCEL  button,  at  the  top  left  side  of  the  screen,  should  be  used  ONLY  if  you  would  like  to 
discard  the  data  you  entered.  ALL  data  related  to  the  current  interview  will  be  lost. 


The  "follow-up"  button  of  the  welcome  screen  will  lead  you  to  the  following  screen. 


You  need  to  enter  the  patient's  id  and  click  the  "Go"  button  to  enable  the  questionnaire.  You 
must  have  interviewed  the  patient  regarding  regret  (Welcome  screen,  "1st  interview"  button) 
before  you  use  this  questionnaire.  You  need  to  answer  all  questions.  Click  the  "Cancel"  button 
to  discard  the  interview.  Click  the  "Save/Exit"  button  to  save  your  work  and  return  to  the 
Welcome  screen. 

Study  recruitment  and  interview  process  (healthcare  provider  and  patients) 

You  will  collect  data  from  healthcare  providers  and  their  (terminally  ill)  patients.  The 
participating  physicians  will  identify  the  "terminally-ill"  patients  who  are  potential  candidates 
for  participation  in  this  study.  Hence,  the  first  step  in  the  data  collection  process  is  the 
recruitment  and  data  collection  from  healthcare  providers. 

Healthcare  provider  interview  guidelines 

Study  recruitment  (healthcare  provider) 

•  We  will  provide  a  short  summary  of  study  procedures  and  expectations  from  the 
physicians  to  all  physicians  affiliated  to  TGH  and  Moffitt  Cancer  Center.  (Via:  email  /  a 
hard  copy/flyer). 


Tasks  to  be  completed  during  the  initial  visit: 


•  Request  the  healthcare  providers  who  are  interested  in  participating  in  the  study  to  sign 
the  informed  consent  form  (appendix  3). 

•  Please  provide  a  photocopy  of  the  signed  informed  consent  form  to  the  participating 
healthcare  providers  during  the  subsequent  visit. 

•  Request  the  full  name  and  the  TGH/  Moffitt  Cancer  Center  identification  numbers  for 
the  patients  that  the  participating  physicians  are  providing  services  for. 

•  Further,  request  the  participating  physicians  to  identify  the  patients  who  are 
categorized  as  "terminally-ill"  (potential  hospice  candidates)  according  to  their  clinical 
judgment  at  the  current  time. 

•  Inform  the  physician  that  you  (or  your  fellow  research  nurse)  will  approach  the  potential 
participants  (terminally-ill  patients  under  his  /  her  care)  in  order  to  request  participation 
in  this  study.  Also,  inform  the  healthcare  provider  that  you  will  see  him  /  her  again  once 
you  have  obtained  data  and  completed  interview  (s)  of  his/  her  terminally-ill  patients. 

•  Obtain  the  demographic  and  other  information  depicted  in  the  healthcare  provider 
demographics  form  (appendix  1). 

•  Thank  the  healthcare  provider  and  make  tentative  appointment(s)  to  see  him/her  again. 

Tasks  to  be  completed  during  the  interview  (second  visit): 

•  Please  note  that  you  will  be  using  both  the  paper  form  and  the  Hospice  Referral 
software  to  collect  data  and  interview  details. 

•  Obtain  the  healthcare  providers'  prediction  of  survival  for  the  patient  being  discussed 
and  note  in  the  physician  EBM  CDSS  usefulness  case  report  form  (appendix  6). 

•  Start  the  Hospice  Referral  software. 

•  On  the  Welcome  screen  (Figure  1)  click  on  the  "1st  Interview"  Button.  This  will  lead  you 
to  the  main  interview  window  of  the  Hospice  Referral  software. 


Figure  1.  Welcome  screen  of  the  Hospice  Referral  Software 

Start  the  interview  by  providing  the  name  and  TGH/Moffitt  Cancer  Center  ID  of  the 
patient  you  are  going  to  discuss  about. 

Use  the  Hospice  Referral  software  to  record  the  Physician's  ID  and  the  patient's  ID  (see 
figure  2). 

In  the  "Who  is  interviewed"  dropped  down  menu  select  "Physician". 

Notice  that  the  "Interview  type"  as  well  as  the  "Familiarity  with  hospice  services"  boxes 
are  deactivated.  The  "Interview  Type"  box  corresponds  to  the  level  of  the  interview  the 
interviewee  agrees  to  participate  in  (e.g.  full:  access  to  medical  records  and  regret 
interview;  partial:  access  to  medical  records;  and  minimum:  access  to  date  of  death). 
Healthcare  providers  are  participating  only  to  full  type  interviews.  The  "Familiarity  with 
hospice  services"  box  is  used  to  capture  the  patient's  awareness  of  hospice  care. 
Therefore,  it  is  not  applicable  to  healthcare  providers. 


Figure  2.  Initial  interview  screen  of  the  Hospice  Referral  software:  Use  this  tab  to  enter  the 

interview  details. 

•  The  NEXT  button  from  figure  2  will  lead  you  to  the  "Current  Choice"  tab,  displayed  in 
figure  3. 


Figure  3.  Current  Choice:  Use  this  tab  to  enter  what  the  healthcare  provider  believes  is  the  best 

action  for  the  particular  patient. 

•  Use  the  Hospice  Referral  software  to  record  what  the  healthcare  provider  believes  is  the 
best  treatment  option  for  the  particular  patient  by  clicking  the  corresponding  radio 
button. 

•  Click  the  NEXT  button  of  the  "Current  Choice"  tab  (figure  3). 

•  The  next  tab  will  be  the  "Regret"  (figure  4)  tab.  Click  the  "next"  button  to  go  to  the 
regret  tab  (figure  3). 


Figure  4.  Regret  Tab:  Use  this  tab  to  enter  the  healthcare  providers'  regret  level  regarding  each 

treatment  option  for  the  particular  patient. 

•  Use  the  script  provided  to  elicit  the  healthcare  providers'  regret  levels  (appendix  2). 

•  Use  the  Hospice  Referral  software  to  record  the  healthcare  providers'  regret  levels  by 
moving  the  corresponding  sliders  depicted  in  figure  4. 

•  Read  the  summary  statement  at  the  bottom  of  the  "Regret"  tab  (figure  4  -"Summary" 
box)  to  validate  the  healthcare  providers  choice.  If  the  healthcare  provider  does  not 
agree  with  this  statement,  ask  him/her  to  re-evaluate  his/her  regret  estimates. 

•  You  are  now  ready  to  move  to  the  next  tab  (figure  5  -  "Summary")  by  clicking  the  NEXT 
button  depicted  in  figure  4. 

•  The  next  tab  in  the  Hospice  Referral  software  presents  information  regarding  the 
assessment  of  our  statistical  model  and  decision  methodology  (figure  5).  If  the 


healthcare  provider  wishes  to  learn  our  assessment  as  well  as  the  probability  of  death 
for  the  patient,  offer  the  information  presented  in  this  tab. 


Figure  5.  Summary  tab:  Use  this  tab  to  inform  the  healthcare  provider  regarding  the  details  of 

our  assessment. 

•  You  may  click  the  button  "Show  Chart"  in  the  "Summary  tab"  to  show  a  graphical 
representation  of  the  survival  probability  computed  by  the  software  if  the  healthcare 
provider  wishes  to  see  it.  The  graphical  representation  appears  as  follows  (figure  6): 


Figure  6.  Graphical  representation  of  survival  probability. 

You  can  return  to  the  "Summary"  tab  by  clicking  the  "Return"  button. 

You  can  now  continue  to  the  next  tab  "Questionnaire"  by  clicking  the  Next  button  in  the 
"Summary"  tab  (figure  5) 

The  "Questionnaire"  tab  contains  questions  regarding  the  usefulness  of  our  statistical 
model  in  healthcare  providers'  decision-making  (figure  7) 

Ask  the  healthcare  provider  if  he/she  would  like  to  change  his  mind  and  therefore  re¬ 
evaluate  his  regret  levels. 

If  the  healthcare  provider  wishes  to  change  his/her  mind,  click  on  the  "Yes"  button  on 
the  top  of  the  "Questionnaire"  tab.  By  clicking  the  "Yes"  button,  a  new  record  will  be 
created  and  the  Hospice  Software  will  return  to  the  "Regret"  tab  (figure  4). 

o  Repeat  the  procedure  until  you  reach  an  agreement  with  the  healthcare  provider 
(he/she  will  not  change  his/her  mind) 

If  the  healthcare  provider  does  not  wish  to  change  his/her  mind,  click  on  the  button 
"No".  This  will  enable  the  questionnaire  for  filling  out.  To  answer  each  question,  click  on 
the  radio  button  that  represents  the  healthcare  provider's  answer. 


Figure  7.  Questionnaire  tab:  Use  this  tab  to  get  the  healthcare  provider's  reaction  to  our 
recommendation.  In  this  tab,  the  healthcare  provider  has  the  opportunity  to  change  his/her 

mind  regarding  the  regret  he/she  feels. 

•  When  the  physician  answers  all  questions  in  the  "Questionnaire"  tab,  you  can  move  to 
the  next  tab  by  clicking  the  "Next"  button  in  the  bottom  of  the  "Questionnaire"  tab 
(figure  7). 

•  The  next  tab  you  will  go  to  will  be  the  "Acceptable  Regret"  (figure  8)  tab. 


Figure  8.  Acceptable  Regret  tab:  Use  this  tab  to  record  the  healthcare  provider's  acceptable 

regret 

•  Use  the  acceptable  regret  script  for  elicitation  of  healthcare  providers'  acceptable  regret 
(appendix  2).  Record  the  healthcare  providers'  values  in  the  appropriate  boxes  (figure  8) 

•  You  can  now  move  to  the  last  tab  "Final"  (figure  9)  by  clicking  the  "Next"  button  at  the 
bottom  of  the  "Acceptable  Regret"  tab  (figure  10). 


Figure  10.  Final  tab:  Use  this  tab  to  save/cancel  your  work. 

The  "final"  tab  reminds  you  to  thank  the  healthcare  provider  and  save  your  work.  Click 
on  the  "Save"  button  on  the  Hospice  Referral  software  (figure  10).  This  will  lead  you 
back  to  the  Welcome  screen. 

Repeat  these  steps  for  each  terminally  -ill  patient  (whom  you  or  your  fellow  research 
nurse  have  interviewed  earlier)  receiving  care  with  the  healthcare  provider  being 
interviewed. 


Thank  the  healthcare  provider  and  conclude  the  interview. 


Patient  interview  guidelines 


Study  recruitment  (patients) 

•  Please  contact  potential  patients  (and  /  or  their  family  members)  in  person. 

•  Please  explain  the  purpose  of  the  study  using  the  recruitment  script  (which  is  a  short 
summary  of  study  procedures  and  expectations  from  the  participants:  appendix  1)  to  all 
the  potential  participants. 

Tasks  to  be  completed  during  the  interview: 

•  After  you  have  explained  the  study  purpose  and  other  details  using  the  recruitment 
script,  ask  if  the  participant  needs  more  time  to  think  and  wants  you  to  come  at  later 
time  etc.  Exercise  maximum  tact,  compassion  and  understanding. 

•  Please  request  patients  who  are  interested  in  participating  in  the  study  to  sign  the 
informed  consent  form  (appendix  3). 

•  Please  provide  enough  time  for  the  patient  to  read  and  understand  the  informed 
consent  form.  Also,  inform  the  patients  that  you  will  be  ready  to  answer  any  questions  / 
concerns  they  may  have. 

•  For  the  patients  who  are  not  in  a  state  of  being  able  to  sign  the  informed  consent  form: 
request  member  of  patients'  family  (a  proxy  /  surrogate  member)  to  sign  the  proxy 
informed  consent  form  (appendix  3). 

•  Inform  the  patient  that  you  will  provide  a  photocopy  of  the  signed  informed  consent 
form  during  the  subsequent  visit. 

•  Inform  the  patient  that  you  will  be  extracting  laboratory  data  from  his/her  hospital 
charts  to  calculate  life  expectancy. 

•  Please  complete  the  demographics  and  other  information  in  the  patient  case  report 
form  table  1  (appendix  1). 

•  Also,  please  obtain  and  complete  data  required  in  the  tables  2,  3  and  4  in  the  patient 
case  report  form.  Please  use  the  patients'  hospital  charts  and  other  hospital  records  to 
obtain  these  data.  Contact  the  staff  nurse  if  required. 

•  Thank  the  patient  for  his  participation  in  the  study  and  continue  with  the  interview. 


For  patients  who  agree  to  participate  in  the  complete  study: 

•  Start  the  Hospice  Referral  software. 

•  On  the  welcome  screen  click  on  the  "1st  interview"  button  (figure  11) 


Figure  11.  Welcome  screen  of  the  Hospice  Referral  Software.  Click  on  the  "1st  interview"  button 

for  the  first  interview  with  the  patient/surrogate. 

•  Obtain  the  full  name  and  patient  identification  number  and  write  it  down  on  the  case 
report  form  (appendix  1). 

•  Write  down  the  identification  number  of  the  treating  physician  on  the  case  report  form 
(appendix  1). 

•  Use  the  Hospice  Referral  software  to  record  the  Physician's  and  the  patient's 
identification  numbers  (see  figure  12). 

•  In  the  "Who  is  interviewed"  dropped  down  menu  select  "Patient  or  Surrogate". 

•  If  the  patient  is  not  in  a  state  being  interviewed  and  his  /  her  proxy  has  signed  the 
appropriate  informed  consent  form  please  select  "Surrogate"  from  the  drop  down  menu 
and  proceed  to  interview  the  proxy  individual. 

•  Per  the  informed  consent  form  there  are  three  types  of  interviews  a  patient/surrogate 
may  agree  to  participate  in:  full  (access  to  medical  records  and  regret  interview),  partial 
(access  to  medical  records),  and  minimal  (access  to  the  date  of  death).  Based  on  the 
patient's/surrogate's  response  select  the  appropriate  option  button  in  the  "Interview 
type"  box  (figure  12) 

•  Ask  if  the  patient  is  aware  of  hospice  services  and  if  he  was  approached  by  a  hospice 
nurse.  If  so,  fill  out  the  "Familiarity  with  hospice  services"  box  (figure  12). 


Data  Collection 


Cancel 


Enter  patient's  ID 


Interviewee 
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Patient  ID  PatientID 

Physician  ID  Physician  ID 
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(J  The  interviewee  will  only  allow  us  to  collect  his/her  date  of  death 

Click  to  continue 


Check  the  appropriate  interview 
type 


Figure  12.  Interviewee  tab:  Use  this  tab  to  enter  the  physician's  and  patient's  id  and  to  select 

the  interview  type. 

•  Click  on  the  Next  button  on  the  bottom  of  the  "Interviewee"  tab  (figure  12)  to  move  to 
the  "Prognostic  models"  Tab  (figure  13) 

•  At  the  "Prognostic  models"  tab  enter  the  PPS  Score  and  SUPPORT  details  (figure  13). 
Please  use  the  guidelines  for  calculation  of  PPS  document  (appendix  1). 


Select 


Figure  13.  Prognostic  models:  Use  this  tab  to  enter  physiological  information  regarding  the 

patient. 

•  PPS  Score  is  the  only  required  data  before  you  are  allowed  to  move  the  next  tab. 
However,  use  the  patient's  chart  to  fill  out  the  remaining  information  (figure  13). 

o  Hovering  the  mouse  pointer  on  the  label  of  each  textbox  (e.g.  the  word  "Age", 
"Albumin"  etc.)  an  information  tooltip  will  appear  showing  you  the  range  of 
normal  values  for  each  category.  Use  this  information  to  get  an  estimate  of  the 
values  required. 

o  The  drop  down  menu  regarding  the  presence  of  cancer  related  to  both  the  main 
disease  (e.g.  Colon  cancer)  and  any  comorbidity.  Therefore,  even  if  the  main 
disease  of  the  patient  is  not  related  to  cancer  (e.g.  congestive  heart  failure)  but 
the  patient  has  cancer,  you  should  note  that  in  the  "cancer"  drop  down  menu. 

•  Once  you  complete  entering  the  patient's  information  in  the  "Prognostic  Models"  tab, 
click  on  the  NEXT  button  (figure  13).  If  there  are  missing  values  in  the  prognostic  models 
tab,  you  will  receive  a  notice  as  the  one  shown  in  figure  14.  If  you  have  more  data  to 
enter,  click  "No"  and  continue  entering  the  data  missed.  If  you  have  no  more  data  to 
enter,  click  "Yes" 


o  Note  that  you  will  not  be  able  to  return  to  this  screen  to  enter  missing  values. 
Refer  to  the  Troubleshooting  section  of  this  manual  to  learn  how  to  handle 
missing  values  at  a  later  time. 


Figure  14.  Warning  related  to  missing  data.  Make  sure  you  have  no  more  data  to  enter. 

The  next  tab  is  associated  with  the  "Current  Choice"  of  the  patient/surrogate,  displayed 
in  figure  15. 

o  Use  the  Hospice  Referral  software  to  record  what  the  patient/surrogate  believes 
is  the  best  treatment  option  for  the  particular  patient  by  clicking  the 
corresponding  radio  button.  Please  use  the  regret  script  to  elicit  this  information 
(appendix  2). 


Figure  15.  Current  Choice:  Use  this  tab  to  enter  what  the  patient/surrogate  believes  is  the  best 

action  for  the  particular  patient. 

•  Click  the  NEXT  button  of  the  "Current  Choice"  tab  (figure  15). 

•  The  next  tab  is  the  "Regret"  tab  (figure  16),  in  which  the  interview  regarding  the  regret 
levels  of  the  patient/surrogate  will  be  implemented. 


Figure  16.  Regret  Tab:  Use  this  tab  to  enter  the  patient's/surrogate's  regret  level  regarding  each 

treatment  option  for  the  particular  patient. 

•  Use  the  script  provided  to  elicit  the  patient's/surrogate's  regret  levels  (appendix  2). 

•  Use  the  Hospice  Referral  software  to  record  the  patient's/surrogate's  regret  levels  by 
moving  the  corresponding  sliders  depicted  in  figure  16. 

•  Read  the  summary  statement  in  the  bottom  of  the  "Regret"  tab  (figure  16  -"Summary" 
box)  to  validate  the  patient's/surrogate's  choice.  If  the  patient/surrogate  does  not  agree 
with  this  statement,  ask  him/her  to  re-evaluate  his/her  regret  estimates. 

•  By  clicking  the  "Next"  button  in  the  "Regret"  tab  the  software  will  lead  you  to  the 
"Summary"  tab  (figure  16),  which  presents  details  about  our  assessment  as  well  as  the 
probability  of  death  for  the  patient.  If  the  patient  wishes  to  know  our  assessment  offer 
him/her  the  details  of  this  tab. 

•  If  the  patient  wishes  to  know  more  regarding  his  probability  of  death  you  may  offer  a 
graphical  representation  by  clicking  the  button  "Show  Chart".  This  will  show  the  window 
depicted  in  figure  17. 


Figure  17.  Summary  tab:  Use  this  tab  to  inform  the  patient/surrogate  regarding  the  details  of 

our  recommendation. 


Figure  18.  Graphical  representation  of  probability  of  death 


In  the  chart  window,  click  "Return"  to  return  to  the  "Summary"  tab. 

You  can  now  continue  to  the  next  tab  "Questionnaire"  by  clicking  the  "Next"  button  in 
the  "Summary"  tab  (figure  19) 

The  "Questionnaire"  tab  contains  questions  regarding  the  usefulness  of  our  statistical 
prognostication  model  in  patients'/surrogates'  decision-making  regarding  choosing 
hospice  care  versus  continuing  current  treatment  (figure  19). 

Ask  the  patient/surrogate  if  he/she  would  like  to  change  his/her  mind  and  therefore  re¬ 
evaluate  his/her  regret  levels. 

If  the  patient/surrogate  wishes  to  change  his/her  mind,  click  on  the  "Yes"  button  on  the 
top  of  the  "Questionnaire"  tab  (figure  19).  By  clicking  the  "Yes"  button,  a  new  record 
will  be  created  and  the  Hospice  Software  will  return  to  the  "Regret"  tab  (figure  16). 

o  Repeat  this  procedure  until  the  patient/surrogate  does  not  wish  to  change 
his/her  mind. 

If  the  patient/surrogate  does  not  wish  to  change  his/her  mind,  click  the  "No"  button  to 
activate  the  questionnaire.  To  answer  each  question,  click  on  the  radio  button  that 
represents  the  patient's/surrogate's  answer. 
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Based  on  what  we  have  talked  about  today,  would  you  choose  hospice  care? 
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Figure  19.  Questionnaire  tab:  Use  this  tab  to  get  the  patient's/surrogate's  reaction  to  our 
recommendation.  In  this  tab,  the  patient/surrogate  has  the  opportunity  to  change  his/her  mind 

regarding  the  regret  levels  he/she  feels. 

•  When  the  patient/surrogate  answers  all  questions  in  the  "Questionnaire"  tab,  you  can 
move  to  the  next  tab  by  clicking  the  "Next"  button  at  the  bottom  of  the  "Questionnaire" 
tab  (figure  20). 

•  If  the  patient/surrogate  has  agreed  to  the  acceptable  regret  interview  (figure  13  - 
"Interview  type"),  the  next  tab  will  be  the  "Acceptable  Regret"  (figure  20). 

•  Please  use  the  acceptable  regret  script  for  elicitation  of  patients'  /  surrogates' 
acceptable  regret  (appendix  2). 

•  Please  record  the  patient's  /  surrogate's  values  in  the  appropriate  boxes  (figure  20). 


Figure  20.  Acceptable  Regret  tab:  Use  this  tab  to  record  the  patient's/surrogate's  acceptable 

regret 

•  You  can  now  move  to  the  last  tab  "Final"  (figure  21)  by  clicking  the  Next  button  on  the 
bottom  of  the  "Acceptable  Regret"  tab  (figure  20). 


Figure  21.  Final  tab:  Use  this  tab  to  save/cancel  your  work. 

The  "final"  tab  reminds  you  to  thank  the  patient/surrogate  and  save  your  work.  Click  on 
the  Save  Button  (figure  21). 

Thank  the  patient  and  make  a  tentative  appointment  to  see  him/her  after  one  month 
(the  post-decision  regret  interview  should  take  place  after  he  /  she  makes  his  decision 
regarding  choosing  hospice  care  or  continuing  current  treatment)  to  obtain  the  post¬ 
decision  regret. 

For  each  post-decisional  regret  interview  please  use  the  post-decisional  regret  scale 
(appendix  2). 

o  Start  the  Hospice  Referral  Software 

o  On  the  welcome  screen  click  the  button  "Follow  up"  (figure  22).  This  will  bring 
the  main  questionnaire  for  the  post  regret  interview  (figure  23) 


Figure  22.  Welcome  window  of  the  Hospice  Referral  Software.  Click  on  the  "Follow  up"  button 

for  post  regret  interviews 

o  Enter  the  patient's  ID  on  the  appropriate  box  (figure  23).  Click  the  "Go"  button. 
The  software  will  check  if  you  have  interviewed  the  patient  regarding  regret 
before.  If  so,  the  questionnaire  will  be  enabled  to  fill  out.  If  not,  the  software  will 
inform  you  that  you  need  to  interview  the  patient  about  regret  first. 

o  After  you  fill  out  the  questionnaire,  click  on  the  "Save/Exit"  button  to  save  your 
work  and  return  to  the  "welcome"  window.  Or,  click  on  the  "Cancel"  button  to 
discard  your  entries. 

o  Thank  the  patient  and  make  a  tentative  appointment  to  see  him/her  after  one 
month. 


Figure  23.  Post  regret  interview  questionnaire. 


Troubleshooting  the  Hospice  Referral  Software 


How  to  run  the  Hospice  referral  software? 

To  run  the  Hospice  Referral  software,  you  need  to  locate  the  Hospice  Referral  shortcut  on  the 
computer's  desktop.  Double  click  the  shortcut  to  start  the  software.  If  you  cannot  locate  the 
Hospice  Referral  shortcut,  ask  the  study  personnel  to  re-install  the  software. 

What  are  the  Physician  and  Patient  ID? 

The  physician's  and  patient's  IDs  are  the  coded  numbers  we  have  provided  you  referring  to 
each  physician  and  patient,  respectively.  Remember  that  these  IDs  are  de-identified  and  do  not 
include  names  or  date  of  births.  The  IDs  are  of  the  form:  XXXXXXXX. 

What  happens  if  a  patient  has  more  than  one  physician? 

If  a  patient  has  more  than  one  physician's  then  interview  each  physician  who  consents  for  the 
study. 

What  happens  if  I  do  not  have  all  information  related  to  the  SUPPORT  model? 

If  you  do  not  have  all  information  required  for  the  SUPPORT  model,  then  fill  out  as  much  as  you 
can  before  continuing  to  the  next  tab. 

I  accidentally  pressed  Cancel  and  now  I  lost  my  work.  Can  I  retrieve  my  work? 

You  will  not  be  able  to  retrieve  your  work  once  you  have  clicked  the  "Cancel"  button.  However, 
before  clicking  the  "Cancel"  button  a  warning  message  will  verify  your  action. 

Can  I  edit  a  record  I  already  saved? 

You  will  not  be  able  to  edit  a  record  once  you  have  clicked  the  "Save"  button.  However,  before 
clicking  the  "Save"  button  a  warning  message  will  verify  your  action. 

Can  I  edit  a  record  before  I  save  it? 

Once  you  have  reached  the  "Final"  form  of  your  record  you  may  navigate  backwards  to  any  of 
the  tabs  in  the  Hospice  Referral  software  and  you  can  edit  all  information  entered.  Remember 
that  you  will  need  to  navigate  forward  by  clicking  the  "Next"  button  located  at  the  bottom  of 
each  tab. 

Can  I  navigate  to  previous  tabs? 

You  can  navigate  to  previous  tabs  by  clicking  the  name  tag  of  each  tab.  Use  the  "Next"  button 
at  the  bottom  of  each  tab  to  navigate  forward. 


I  cannot  find  the  Save  button  to  save  my  work 


The  "Save"  button  is  located  in  the  "Final"  tab  of  the  regret  interview  window.  You  need  to 
navigate  through  all  of  the  steps/tabs  of  the  regret  interview  to  reach  the  "save"  button. 

How  can  I  terminate  /  close  the  software? 

Use  the  "Cancel"  button  located  at  the  top-right  side  of  the  Hospice  Referral  software  to  close 
the  application  and  then  click  on  the  "Exit"  button  of  the  Welcome  screen.  Remember:  any 
unsaved  work  will  be  lost. 

After  the  regret  interview,  I  realized  that  I  did  not  enter  all  lab  values  required  in  the 
Prognostication  tab.  Can  I  enter  these  values  at  a  later  time? 

The  values  used  in  the  "Prognostication"  tab  are  necessary  for  the  accurate  estimation  of  life 
expectancy  communicated  during  the  interviews.  Therefore  it  is  imperative  to  record  all  of 
these  values  before  an  interview. 

You  will  NOT  be  able  to  enter  missing  values  after  the  interview.  However,  please  use  the 
appropriate  paper  base  data  collection  forms  to  enter  the  missing  values  and  notify  the 
research  personnel  regarding  the  values  missed  and  the  patient  id  they  correspond  to. 


Appendix  12  Study  flyer 


USF 


HEALTH 


“Hospice  Referral  Study” 

EBM-CDDS  (Evidence  Based  Medicine  -  Clinical  Decision  Support  System) 


Objective: 

To  aid  prognostication  in  terminally  ill  patients  using  evidence  based 

clinical  decision  support  system 

We  want  to  evaluate  the  EBM-CDDS  in  patients  that  are  eligible  for  hospice  referral 


How  can  you  contribute? 

•  Call  our  research  team  for  any  patient  you  think  may  be  eligible  for 
hospice  care. 

How  can  you  benefit? 

•  We  will  assess  your  patient’s  preferences  and  calculate  his  or  her  life 
expectancy  to  advise  the  patient  if  the  referral  to  hospice  is  the  “right 
choice”  for  him  or  her. 

We  are  looking  forward  to  your  participation  in  this 
important  research  endeavor  to  improve 
prognostication  and  patient  referral  to  hospice! 

Contact  Information 

Stephanie  Warburton  Office  (813)  396-9178  swarburt@health.usf.edu 

Cell  (813)  767-3819 

Dr.  Rahul  Mhaskar  Office  (813)  974-9608  rmhaskar@health.usf.edu 

Cell  (765)  337-5899 

Dr.  Benjamin  Djulbegovic  Office  (813)  396-2349  bdjulbeg@health.usf.edu 

Cell  (813)  679-4930 

Study  PI:  Benjamin  Djulbegovic,  MD,  PhD 

Study  funded  by  DOA  grant  #  W81XWH-09-2-0175 
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I.  Overview 


Welcome  to  the  pain  management  module 

The  pain  management  module  (PMM)  has  been  designed  to  aid  physicians  in  clinical 
decision  making.  It  integrates  highest  quality  of  available  evidence  with  NCCN  physician 
guidelines  to  yield  a  comprehensive  pain  management  system.  The  PMM  enables  the  physician 
to  assess  patient's  pain  level,  prescribe  appropriate  pain  medication,  calculate  dosages  for 
opioid  conversion,  manage  adverse  effects  associated  with  pain  medication,  provide 
educational  and  psychosocial  support  resources,  and  easily  access  information  for  all  available 
pain  medications.  The  PMM  aims  to  boost  physician  confidence  in  clinical  decision  making 
while  providing  patients  with  more  effective  pain  management. 

As  it  is  based  on  NCCN  guidelines,  the  PMM  is  primarily  intended  for  use  in  patients 
experiencing  pain  associated  with  cancer.  However,  certain  modules  (i.e.  pain  assessment  and 
opioid  conversion)  can  be  applied  to  managing  'any-source'  pain.  The  NCCN  guidelines  have 
been  supplemented  by  the  highest  quality  of  available  evidence  on  each  drug  through  a 
comprehensive  literature  search. 

Disclaimer 

Any  program  User  is  expected  to  use  independent  medical  judgment  in  the  context  of 
individual  clinical  circumstances  to  determine  patient's  care  or  treatment.  All  calculations  must 
be  confirmed  before  use.  The  authors  make  no  claims  of  the  accuracy  of  the  information 
contained  herein;  and  these  suggested  doses  are  not  a  substitute  for  clinical  judgment. 
University  of  South  Florida  -  Division  of  Evidence-Based  Medicine  and  Health  Outcomes 
Research  nor  any  other  party  involved  in  the  preparation  of  this  program  shall  be  liable  for  any 
special,  consequential,  or  exemplary  damages  resulting  in  whole  or  part  from  any  user's  use  of 
or  reliance  upon  this  material. 

Manual  Structure 

The  manual  is  broken  down  into  three  components. 

1)  The  introduction  will  provide  the  basis  for  the  pain  management  module  as  well  as  the 
theory  of  evidence  based  clinical  decision  support. 

2)  Next,  using  the  main  page  as  a  starting  point,  each  possible  path  will  be  followed  to  the 
end 

3)  Finally,  for  experienced  users,  shortcuts  to  specific  pages  of  the  pain  management 
module  are  provided. 
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Remarks: 


Before  using  this  program,  we  strongly  recommend  User  be  familiar  with  NCCN  guidelines 
for  "Adult  Cancer  Pain". 

All  program  users  are  encouraged  to  try  different  scenarios,  till  they  become  very 
familiar  with  the  program  and  its  features.  Also,  pages  contain  additional  explanations  and 
tables  (show/hide)  and  useful  links.  Please,  try  it. 

By  moving  "mouse"  over  some  of  the  table  fields,  program  will  pop-up  small  windows  with 
instructions.  This  is  useful  in  different  conversions  with  more  than  one  "operation". 


II.  What  is  the  Pain  Management  Module? 

Clinical  decision  support  systems 

Clinical  decision  support  systems  (CDSS)  aim  to  provide  physicians  with  higher 
confidence  in  clinical  decision  making  by  linking  clinical  observations  with  the  best  available 
evidence  in  the  field. 

Ideally  a  CDSS  should  be  informed  by  the  totality  of  evidence.  In  development  of  PMM, 
a  systematic  evaluation  of  all  the  currently  available  web-based  resources  related  to  choice  of 
drug  for  pain  management,  route  of  administration,  conversion  between  drugs  and  dosage  was 
performed.  When  there  was  a  lack  of  published  evidence,  suggestions  from  Package  insert 
were  used  in  providing  dosing  and  administration  routes  for  specific  drugs. 

Pain  management  in  cancer 

Pain  management  is  a  branch  of  medicine  employing  an  interdisciplinary 
approach  for  easing  the  suffering  and  improving  the  quality  of  life  of  those  living  with  pain.  The 
typical  pain  management  team  includes  medical  practitioners,  clinical  psychologists, 
physiotherapists,  occupational  therapists,  and  nurse  practitioners. 

Pain  is  a  common  symptom  experienced  by  patients  with  cancer.  Whether  as  a  result  of 
the  disease  or  a  disease-related  treatment,  pain  causes  significant  physical  and  psychosocial 
burdens. 

NCCN  reported  that  cancer  pain  can  be  well  controlled  in  the  vast  majority  of  patients  if 
evidence-based  guidelines  are  applied,  monitored,  and  individualized  and  patients  engage  in 
informed  decision  making  for  managing  their  pain. 
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NCCN  guidelines 


NCCN  (National  Comprehensive  Cancer  Network)  guidelines  are  considered  one  of  the 
most  reputable  professional  guidelines  and  are  widely  used  to  manage  patients  with  cancer  and 
related  conditions.  NCCN  guidelines  are  developed  by  21  leading  institutions  in  the  US  and  are 
considered  most  reliable  source  of  advice  both  for  patients  and  physicians  alike. [cite] 

NCCN  guidelines  introduce  several  important  components,  such  as:  Pain  intensity 
quantified  by  the  patient  (whenever  possible),  reassessment  of  pain  intensity,  availability  of 
psychosocial  support  and  provision  of  specific  educational  material.  They  also  provide  dosing 
guidelines  for  NSAIDs,  opioids  and  co-analgesic.  Finally,  they  provide  suggestions  for  titrating 
and  rotating  opioids,  changing  dose  and  route  of  administration,  and  managing  of  opioid 
adverse  effect. 
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III.  Main  page 


After  first  opening  the  pain  management  module,  the  user  should  be  able  to  see  the 
screen  below: 


Home  Pain  Rating  ~  Opioid  Conversion  v  Neuropathic  Pain  v  Adverse  Effects  ^  information  on  Drugs  ~  Write  Rx  v 

CDSS-EBM  Pain  Management  Module 
What  would  you  like  to  do? 


Pain  Assessment 


Sub-Modules  for  Pain  Assessment 


Pain  Intensity  Rating 

Opioid  Conversion 

Neuropathic  Pain 

Adverse  Effects 

Information  on  Drugs 

Numerical  rating  icale 

General  conversion 

Antidepressants  Constipation  &  Nausea 

Nonopioid  Analgesic 

Visual  ratine  teal* 

Intravenous  protocol 

Anticonvulsants 

Pruritus  &  Delirium 

Opioid  Analgesic 

Transdermal  patch 

Topical  agents 

Others 

Co-Analgesic 

Corticosteroids 

PiKUiag 

Figure  1 

Instructions 

Some  options  on  this  screen  are  available  on  every  page  of  the  module  and  other 
options  are  specific  to  this  page.  First  we  will  review  options  found  on  every  page  of  the 
module. 
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Options  common  to  all  pages  of  the  pain  management  module: 


Home  Pain  Rating  v  Opioid  Conversion  v  Neuropathic  Pain  v  Adverse  Effects  v  Information  on  Drugs  v  Write  Rx  v 
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4 


5 


Figure  2 
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(1)  Home 

By  selecting  this  option  the  user  will  be  brought  back  to  the  first  screen  of  the  pain 
management  module  (figure  1,  page  7). 

(2)  Pain  Rating 

By  selecting  this  option  the  user  will  be  taken  to  the  pain  intensity  level  assessment 
module  (figure  11,  page  14). 

(3)  Opioid  Conversion 

By  selecting  this  option  the  user  will  be  guided  through  calculating  opioid  conversion 
(figure  54,  page  34). 

(4)  Neuropathic  Pain 

By  selecting  this  option  the  user  will  be  able  to  prescribe  medication  for  management  of 
neuropathic  pain  (figure  56,  page  36). 

(5)  Adverse  Effects 

By  selecting  this  option  the  user  will  be  able  to  prescribe  appropriate  medication  for 
managing  common  adverse  effects  associated  with  pain  medication  (figure  58,  page  37). 

(6)  Information  of  Drugs 

By  selecting  this  option  the  user  will  be  able  to  view  specific  information  on  available 
pain  medications  (figure  60,  page  39). 

(7)  Write  Rx 

By  selecting  this  option  the  user  will  have  options  to  prescribe  medication  using 
suggested  dose  or  to  manually  write  complete  prescription. 


8 


Options  specific  to  the  main  page  of  the  pain  management  module 


The  pain  management  module  interface  consists  of  the  Pain  assessment  module  (1) 
and  five  sub-modules. 


Home  Pain  Rating  ^  Opioid  Conversion  ^  Neuropathic  Pam  ^  Adverse  Effects  ^  information  on  Drugs  ^  Write  Rx  ^ 

CDSS-EBM  Pain  Management  Module 
What  would  you  like  to  do? 


Numerical  rating  scale 
Visual  rating  scale 


General  conversion 
Intravenous  protocol 
Transdermal  patch 


Antidepressants 
Anticonvulsants 
Topical  agents 
Corticosteroids 


Constipation  &  Nausea 
Pruritus  &  Delirium 
Others 


Dticlirg.tr 


Nonopioid  Analgesic 
Opioid  Analgesic 
Co-Analgesic 


Figure  3 


Instructions 


(1)  Pain  Assessment  module 

This  module  contains  all  components  of  the  pain  management  tool.  By  selecting  this 
option,  user  will  be  guided  through  pain  assessment,  patient  drug  history,  and  management 
recommendations. 

There  are  practically  "unlimited  possibilities"  for  how  to  treat  and  follow-up  with 
patients  with  chronic  pain  using  the  "Pain  Assessment"  algorithm. 

Doctors  are  encouraged  to  try  as  much  as  possible  and  attempt  different  scenarios  until 
they  become  very  familiar  with  the  program  and  its  features. 
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(2)  Pain  Intensity  rating  sub-module 

This  sub-module  contains  the  pain  intensity  rating  scales.  By  selecting  this  option,  the 
user  will  be  guided  through  assessment  of  patient's  pain  intensity  using  techniques  provided  by 
NCCN  guidelines.  (See  page  33) 


(3)  Opioid  Conversion  sub-module 

This  sub-module  contains  the  opioid  conversion  calculator.  By  selecting  this  option,  the 
user  will  be  able  to  calculate  dosage  conversions  between  different  opioid  drugs  and  routes  of 
administration.  (See  page  34) 


(4)  Neuropathic  Pain  sub-module 

This  sub-module  contains  management  options  for  neuropathic  pain.  (See  page  35) 


(5)  Adverse  Effects  sub-module 

This  sub-module  contains  options  for  management  of  common  adverse  effects 
associated  with  pain  medication.  (See  page  37) 

(6)  Information  on  Drugs  sub-module 

This  sub-module  contains  information  on  available  pain  medications.  (See  page  39) 
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User  Manual  and  Disclaimer 


In  order  to  access  the  user  manual  and  disclaimer  for  the  pain  management  module 
select  the  appropriate  option  as  shown  in  figure  4. 


Home  Pain  Rating  ~  Opioid  Conversion  v  Neuropathic  Pain  v  Adverse  Effects  ^  information  on  Drugs  ~  Write  Rx  v 

CDSS-EBM  Pain  Management  Module 
What  would  you  like  to  do? 


Pain  Assessment 


Sub-Modules  for  Pain  Assessment 


Pain  Intensity  Rating 

Opioid  Conversion 

Neuropathic  Pain 

Adverse  Effects 

Information  on  Drugs 

Numerical  ratine  scale 
Visual  rating  scale 


Disclaimer 


General  conversion  Antidepressants 

Intravenous  protocol  Anticonvulsants 

Transdermal  patch  Topical  agents 

Corticosteroids 


Figure  4 


Constipation  &  Nausea 
Pruritus  &  Delirium 
Others 


Nonopioid  Analgesic 
Opioid  Analgesic 


User  Manual 
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IV.  Pain  assessment 


The  pain  assessment  module  contains  all  components  of  the  pain  management  tool.  It 
provides  the  user  with  complete  pain  management  option  starting  from  pain  assessment  and 
patient  drug  history  followed  by  management  recommendations  and  prescription  writing. 

To  access  the  Pain  Assessment  module  select  Pain  Assessment  as  shown  in  figure  5. 


Home  Pam  Rating  ^  Opioid  Conversion  ^  Neuropathic  Pain  ^  Adverse  Effects  ^  information  on  Drugs  v-  Write  Rx  ^ 


CDSS-EBM  Pain  Management  Module 


Visual  rating  scale 

DiK-lstnw 


Intravenous  protocol 
Transdermal  patch 


Anticonvulsants 
Topical  agents 
Corticosteroids 


Pruritus  &  Delirium 
Others 


Opioid  Analgesic 
Co-Analgesic 


Figure  5 


Ir.»trjaior.s 


This  will  take  you  to... 

CDSS-EBM  Pain  Management  Module 


Please,  choose  option 

^3)Screen“lS  for  pai° 

Reassess  efficacy  for  previously  seen  the  p; 

Figure  6 

atient 

Let's  select  option  "Screening  for  pain",  and  on  next  page  we  will  have  screen: 
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CDSS-EBM  Pain  Management  Module 


Screening  for  pain 

Is  the  patient  currently 
experiencing  pain? 

No 

No,  but  painful  events  or  procedures  are  anticipa 

Figure  7 

ted. 

Let's  select  option  "Yes",  and  on  next  page  let's  choose  "No". 

CDSS-EBM  Pain  Management  Module 


Assessment 

Is  the  pain  related  to  oncologic  emergency? 


Figure  8 


On  next  page,  let's  choose  "Opioid  naive  patient" 

CDSS-EBM  Pain  Management  Module 


What  is  the  patient  status?  nane  P3**®”* 

Opioid  tolerant  patient 


Figure  9 


After  selecting  "Opioid  naive  patient",  next  screen  will  appear: 

CDSS-EBM  Pain  Management  Module 


Opioid  naive  patients 

Please  choose  option 

)Assess  pain  intensity 

Skip  pain  intensity  assessment  and  go  to  pain  l 

Figure  10 

Management 

By  choosing  "Assess  pain  intensity",  program  will  bring  us  to  the  page  where  we  can 
define  pain  level  for  patient.  There  are  two  options  for  doing  that,  using  numerical  scale 
(clicking  corresponding  radio  button  and  "clicking"  submit  button)  or  with  face  detection  for 
nonverbal  patient  (moving  slider  to  right,  until  face  match  pain  intensity  and  "clicking"  submit 
button). 
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CDSS-EBM  Pain  Management  Module 


show  “hide  Nonverbal  patient  show  hide  Linguistic  assessment 

Pain  Intensity  Rating  -  Naive  Patients 

On  scale  0  to  10,  how  would  you  describe  your  pain?  (You  can  choose  numerical  or  face  description) 

null  one  two  three  four  five  six  seven  eighC  hine  ten 

0  =  no  pain:  1-3  =  mild  pain;  4-6  =  moderate  pain;  7-10  severe  pain  * 

f  Submit 

On  next  page,  you  have  to  confirm  your  selection. 

CDSS-EBM  Pain  Management  Module 


After  confirmation,  you  have  to  choose  route  of  drug  administration  for  short-acting 
opioids.  Let's  for  learning  purpose,  choose  "Tablets".  Choosing  Oral  solution  or  Intravenous, 
there  are  similar  procedures  like  for  Tablets. 
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C’DSS-EBM  Pain  Management  Module 


Opioid  naive  patient  •  routes 

Route  of  drug  administration 

Figure  13 

^O^Tablets  (peak  effect  in  60  minutes) 

Oral  solution  (peak  effect  in  60  minutes) 

Intravenous  (peak  effect  in  15  minutes) 

CDSS-EBM  Pain  Management  Module 


Medication:  (Opioid  tablets) 

Available  dosage: 

▼J  see  opioid  equivalency 

- 

Choose  a  total  daily  dose:  ▼ 

1  day  supply: 

Single  dose: 

7  day  supply:  ° 

30  day  supply: 

If  single  dose  required  "crushed"  tablets,  program  will  set  lower  dose  available  with  tablet  sire. 

After  generating  prescription,  click  here  for  more  prescribing  option,  or  go  back  to  home  page 


Figure  14 


You  have  to  make  three  selections  before  generate  prescription.  You  have  to  make 
medication  selection,  total  daily  dose  and  option  for  supply  (1,  7  or  30  day).  Button  "Generate 
Prescription"  will  be  "alive"  only  if  you  make  correct  selections. 


CDSS-EBM  Pain  Management  Module 


Medication:  (Opioid  tablets) 

Available  dosage: 

Morphine  Sulfate  (^J^Tse^  opioid  equivalency 

Tablets:  15,  30  mg  IR,  Typical  * 

initial  dose  is  15-60  mg,  q3-4h  as 

Choose  a  total  daily  dose:  90  r^g  ▼ 

1  day  supply:  6  (six)  tablets 

Single  dose:  15  mg  tabs 

7  day  supply:  42  (forty  two)  tablets  ° 

Take  single  tablet  every  4  hours,  as  needed 

30  day  supply:  180  (one  hundred  eighty)  tablets 

Generate  Prescription  If  single  dose  required  "crushed"  tablets,  program  will  set  lower  dose  available  with  tablet  size. 

After  generating  prescription,  click  here  for  more  prescribing  option,  or  go  back  to  home  page 


Figure  15 
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CDSS-EBM  Pain  Management  Module 


Medication:  (Opioid  tablets) 

Available  dosage: 

Morphine  Sulfate^)  ▼  see  opioid  equivalency 

Tablets:  15,  30  mg  IR,  Typical  * 

initial  dose  is  15-60  mg,  q3-4h  as  ^ 

Choose  a  total  daily  dose:  120  mg  ▼ 

1  day  supply:  8  (eight)  tablets 

^Single  dose:  15  mg  tabs^> 

7  day  supply:  56  (fifty  six)  tablets 

Takgjswgle-tablot^^^3  hours,  as  needecP^) 

30  dav  supply:  240  (two  hundred  forty)  tablets^}  !  °  1 

Generate  Prescription  If  single  dose  required  ’crushed''  tablets,  program  will  set  lower  dose  available  with  tablet  size. 

After  generating  prescription,  click  here  for  more  prescribing  option,  or  go  back  to  home  page 


Figure  16 


After  making  necessary  selection  (most  of  them  have  default  value),  you  need  to  hit 
"Generate  Prescription"  button  and  on  next  page  you  will  have  prescription  page. 


12901  Bruce  B.  Dorms  Blvd.,  MDC  33 
Tampa,  FL  33612 

Phone: 
0  ME# 


Name:  Saturday,  June  04,  2011 

Rx:  Morphine  Sulfate,  Single  dose:  15  mg  tabs 
Dosage:  Take  single  tablet  every  3  hours,  as  needed 
Dispense:  Month  supply,  240  (two  hundred  forty)  tablets 


Refill:  0=13  1=0  2=0  3=0  4=0  PRN=Q  DEA# 


There  are  two  options.  One  is  to  send  e-mail  direct  to  Pharmacist  or  to  print  prescription 
and  give  patient.  When  we  finish  with  testing  module,  there  will  be  option  that  doctors 
automatically  select  his  name  and  in  same  time  program  will  select  Phone  number,  and 
numbers  for  ME#  and  DEA#.  After  finishing  with  prescription,  we  need  to  close  prescription 
screen. 
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£  prescription  -  Windows  Internet  Explorer 


|€_,  htt  localhost  ▼ 


;  '1  '•  ~  WebS 


Web  Search 


X  ^  Convert  ▼  g  Select 


Favorites  |  ^  0  Suggested  Sites  ▼  Web  Slice  Gallery  ▼ 

&  prescription 


^  - 


El  v  3  fSJD  ▼  Page  ▼  Safety  ▼  Tools  ▼ 


X  USF  Health 


12901  Bruce  B.  Do*™  Btvd.,  MDC  33 
Tampa,  FL  33612 


□ 


Phone: 

ME* 


Name: 

Rx:  Morphine  Sulfate.  Single  dose:  15  mg  tabs 
Dosage:  Take  single  tablet  every  3  hours,  as  needed 
Dispense:  Month  supply,  240  (two  hundred  forty)  tablets 


Refill:  0=0  1=0  2=0  3=0  4=0  PRN=0 

Send  prescription  to  Pharmacist 


Saturday,  June  04,  2011 


DEA* 


Print  this  page 


Figure  18 


C|L  Local  intranet  |  Protected  Mode:  Off  'ffl  'r  %  100% 


We  will  be  back  to  following  screen  (shown  on  figure  16): 


C’DSS-EBM  Pain  Management  Module 


Medication:  (Opioid  tablets) 

Available  dosage: 

Morphine  Sulfate  p]  see  opioid  equhalencv 

Tablets:  15,  30  mg  IR,  Typical  A 

initial  dose  is  15-60  mg,  q3-4h  as 

Choose  a  total  daily  dose:  120  mg 

▼ 

1  day  supply:  8  (eight)  tablets 

Single  dose:  15  mg  tabs 

7  day  supply:  56  (fifty  six)  tablets 

Take  single  tablet  every  3  hours,  as  needed 

30  day  supply:  240  (two  hundred  forty)  tablets  ° 

Generate  Prescription 

If  single  dose  required  "crushed"  tablets,  program  will  set  lower  dose  available  with  tablet  sire. 

After  generating  prescription^  click  here  for  more  prescribing  option,  or  go  back  to  home  page 


Figure  19 
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By  selecting  "click  here"  on  figure  19,  next  screen  will  appear: 


CDSS-EBM  Pain  Management  Module 


Common  to  all  pain  level 

( 

Select  option: 

Prescriptions  for  neuropathic  pain  (co-analgesics  adjuvant) 
Prescriptions  for  opioid  adverse  effects 

Pro\ide  psychosocial  support,  including  patient  and  family  ec 

Figure  20 

ucation 

There  are  options  for  prescription  for  neuropathic  pain,  prescription  for  opioid  adverse 
effect  and  psychosocial  support.  For  learning  purpose,  let's  choose  neuropathic  pain.  Next 
screen  will  show  up: 


CDSS-EBM  Pain  Management  Module 


Co-analgesics  (adjuvant)  medications  for  neuropathic  pain 

c 

Select  co-analgesic  (adjuvant)  medications 

see  principles 

Antidepressants 

Antic  omuls  ants 

Topical  agents 

Corticosteroids 

Figure  21 

Let's  choose  "antidepressants".  Next  screen  will  appear: 

CDSS-EBM  Pain  Management  Module 


Antidepressants: 

Available  dosage  for  selected  antidepressants: 

'▼Vsee)  example 

<•> 

Choose  a  total  daily  dose:  ▼ 

1  day  supply: 

Single  dose: 

7  day  supply:  ® 

30  day  supply: 

Figure  22 

Choosing  one  antidepressants,  next  screen  will  show  up  with  default  setting.  There  are 
possibilities  to  change  daily  dose  and  how  many  days  of  supply  you  want  to  prescribe. 
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CDSS-EBM  Pain  Management  Module 


After  making  necessary  selections,  hit  "Generate  Prescription"  button.  Next  screen  will 
show  up: 


12901  Bruce  B.  Dov.-ns  Blvd.,  MDC  33 
Tampi,  FL  33612 

Phone: 
□  ME  # 


Name:  Saturday,  June  04,  2011 

Rx:  Amitriptyline,  Single  dose:  10  mg  tablet 
Dosage:  Take  single  dose  at  bedtime 
Dispense:  Seven  day  supply,  7  (seven)  tablets 


Refill:  0=0  1=0  2=0  3=0  4=0  PRN=0  DEA* 


Explanation  for  figure  24  is  same  like  for  figure  17  (see  page  16). 
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Let's  close  prescription  page  (figure  25)  and  Antidepressants  page  (figure  26). 
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^  Antidepressants  -  Windows  Internet  Explorer 


^  ^  http:  localhost  correct  %20-%20Benl  ▼  |  X 


Wlrsl  x 


x  c?  Select  H'ab'e  dosage: 

Safety  ▼  Tools  ▼  ▼ 

Favorites  |  <•&  0  Suggested  Sites  ▼  Web  Slice  Gallery  ▼ 

Qt  ▼  0  ▼  □  m  "  Page  ▼ 

'iS  Antidepressants 

Home  Pain  Rating  Opioid  Conversion  Neuropathic  Pam  Adverse  Effects 

Information  on  Drugs 

Write  Rx 

CDSS-EBM  Pain  Management  Module 

Antidepressants: 

Available  dosage  for  selected  antidepressants: 

10,  25,  50,  75,  100,  200  mg 

A 

Amitriptyline  F^]  see  example 

cablets;  Half-life:  10-50  h; 

Er 

Typical  initial  dose:  10-25  mg 

Choose  a  total  daily  dose:  10  mg 

□ 

1  day  supply:  1  (one)  tablet 

Single  dose:  10  mg  tablet 

7  day  supply:  7  (seven)  tablets  '•> 

Take  single  dose  at  bedtime 

30  day  supply:  30  (thirty)  tablets 

_  ° 

Figure  26 

|l  Generate  Prescnption  ]) 

We  will  be  back  to  following  screen  (shown  on  figure  19,  now  marked  like  figure  27)): 


CDSS-EBM  Pain  Management  Module 


Medication:  (Opioid  tablets) 

Available  dosage: 

Morphine  Sulfate  [▼!  see  opioid  equivalency 

Tablets:  15,  30  mg  IR,  Typical  * 

initial  dose  is  15-60  mg,  q3-4h  as  ^ 

Choose  a  total  daily  dose:  120  mg  !_y  j 

1  day  supply:  8  (eight)  tablets 

Single  dose:  15  mg  tabs 

7  day  supply.  56  (fifty  six)  tablets 

Take  single  tablet  every  3  hours,  as  needed 

30  day  supply:  240  (two  hundred  forty)  tablets  ° 

Generate  Prescription  If  single  dose  required  "crushed”  tablets,  program  will  set  lower  dose  available  with  tablet  size. 

After  generating  prescription;  click  here  for  more  prescribing  option;  or  go  back 

By  clicking  "home  page"  program  will  return  to  home  page,  shown  on  figure  28. 
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CDSS-EBM  Pain  Management  Module 
What  would  you  like  to  do? 


Pain  Assessment 


Sub-Modules  for  Pain  Assessment 


Pain  Intensity  Rating 

Opioid  Conversion 

Neuropathic  Pain 

Adverse  Effects 

Information  on  Drugs 

Numerical  ratine  scale  General  conversion 

Visual  rating  scale  Intravenous  protocol 

Transdermal  patch 


Dts-:Uitr.er 


Antidepressants 
Anticonvulsants 
Topical  agents 
Corticosteroids 


Constipation  &  Nausea  Nonopioid  Analgesic 

Pruritus  &  Delirium  Opioid  Analgesic 

Others  Co-Analgesic 


Figure  28 


Instructions 


Let's  go  back  to  figure  10  (see  page  13).  If  you  don't  want  to  go  through  "Assess  pain 
intensity",  because  you  know  pain  intensity,  you  should  choose  second  option  "Skip  pain 
intensity  ..."  This  option  you  can  see  on  figure  29. 


CDSS-EBM  Pain  Management  Module 


Opioid  naive  patients 

Please  choose  option 

Assess  pain  intensity 

Figure  29 

Skip  pain  intensity  assessment  and  go  to  pain  management 

After  selecting  "Skip  pain  intensity  ..."  next  screen  will  show  up  (figure  30). 

There  are  several  options,  but  let's  say  our  patient  has  severe  pain  and  we  would  like  to 
prescribe  short-acting  opioid.  In  this  case  you  should  make  selection  shown  on  figure  30. 

We  are  encourage  users  to  try  all  different  options  and  "familiarized"  with  program. 
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CDSS-EBM  Pain  Management  Module 


Management  of  pain  in  opioid  naive  patient 

For  ALL  pain  levels 

Recognize  and  treat  adverse  effects,  including  bowel  obstruction 

Prescribe  co-analgesics  (adjuvant  medications) 

G  Proride  psychosocial  support,  including  patient  and  family  education 

Severe  Pain  7-10 

See  mangement  for  ALL  levels  of  pain  above,  and 

Rapidly  titrate  short- acting  opioid  with  opition  for  opioids  conversion 

Severe  pain,  full  list  of  strong  agonists 

Moderate  Pain  4-6 

See  mangement  for  ALL  levels  of  pain  above,  and 

Rapidly  titrate  short- acting  opioid  with  opition  for  opioids  conversion 

Moderate  pain  medications,  including  weak  agonists 

Mild  Pain  1  -  3 

1 

See  mangement  for  ALL  levels  of  pain  above,  and 

Prescribe  NSAED  (Modify  regimen  to  minimize  adverse  effects) 

Prescribe  short-acting  opioid  Figure  30 

After  anv  treatment.  Reevaluate  patient's  goals  of  comfort  and  function  at  each  contact 

After  making  selection  on  figure  30,  next  screen  will  show  up  (figure  31). 

CDSS-EBM  Pain  Management  Module 


Short  acting  opioids  -  opioid  naive  patient 

Route  of  drug  administration 

[C^ablets  (peak  effect  in  60  minutes) 

Oral  solution  (peak  effect  in  60  minutes) 

Intravenous  (peak  effect  in  15  minutes) 

Additional  details  of  opioid  principles,  prescribing,  titration,  and  maintenance 

Figure  31 

Figure  31  is  very  similar  to  figure  13  (from  "Assess  pain  intensity").  We  can  choose  same 
option  "Tablets",  and  we  will  have  exactly  same  page  (figure  32)  like  before  (shown  on  figure 
14). 
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CDSS-EBM  Pain  Management  Module 


Medication:  (Opioid  tablets) 

Available  dosage: 

▼  see  opioid  equivalency 

Choose  a  total  daily  dose:  ▼ 

1  day  supply: 

Single  dose: 

7  day  supply:  ° 

30  day  supply: 

If  single  dose  required  "crushed”  tablets,  program  will  set  lower  dose  available  with  tablet  size. 

After  generating  prescription,  click  here  for  more  prescribing  option,  or  go  back  to  home  page 


CDSS-EBM  Pain  Management  Module 


Short  acting  opioids  -  opioid  naive  patient 


Tablets  (peak  effect  in  60  minutes) 

Route  of  drug  administration  t  Oral  solution  (peak  effect  in  60  minutes) 

Intrav  enous  (peak  effect  in  15  minutes) 


^Additional  details  of  opioid  principles,  prescribing,  titration,  and  maintenance 


Figure  33 


Making  selection  like  on  figure  33,  next  (figure  34)  screen  will  show  up: 


CDSS-EBM  Pain  Management  Module 

Opioid  principles,  prescribing,  titration,  and  maintenance 
General  principles  show  hide 

Principles  of  maintenance  opjoid  therapy  show  hide 
("^^Opioid  conversion,  prescribing,  titration,  and  maintenance 


Figure  34 


There  are  information  on  general  principles  and  opioid  therapy,  but  in  the  case  that  you 
want  to  make  any  conversion  between  opioids,  you  have  to  make  selection  shown  on  figure  34. 
Next  screen  will  appear  (figure  35). 
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CDSS-EBM  Pain  Management  Module 


Conversion  GRADE  evidence  profile 

Select  conversion  option: 

show  hide  references  ( 

General  conversion  *!  3?  5 

Between  intravenous  (TV)  medications  6r  7 

Intravenous  (TV)  to  oral  medications  (PO) 

Multiple  PO  and  fentanyl  patch  convert  to  intravenous  me 
Between  oral  medications  with  rescue  doses 

Opioids  to  fentanyl  transdermal  patch  (Duragesic)  4 

dications 

Figure  35 

Let's  make  selection  like  on  figure  35.  After  "clicking"  radio  button,  next  screen  will 
show  up  (figure  36): 


CDSS-EBM  Pain  Management  Module 

Current  medications: 

Oral  opioid  1  3  Daily  dose: 


Oral  opioid  2  ▼]  Daily  dose: 


Intravenous  (TV)  Patient  Controlled  Analgesia  (PCA) 


IV  PCA  [▼]  Concentration:  a 

Basal  Rate:  mg  h  Recommended 


On-Demand:  mgdose  Recommended 

Loading  Dose:  3  Recommended 


Lockout  Interv  al:  ▼ 

Recommended 

*  Product  irueit  for  Dungesic  (Fenaavl  trar.sdermi!  patch)  us*  conservative  conversion  sethod  50ag  PO  morphine  equal  1  Omg  IV  morphine 


Figure  36 


mg  day 
mg  day 


We  will  explain  this  table  more  details  in  our  example  C  (see  page  48),  on  the  end  this 
instruction  manual. 

For  previously  seen  the  patient,  after  pushing  button  "Pain  Assessment"  on  figure  5  (see 
page  12),  next  screen  will  appear  (figure  37):  Let's  this  time  select  "Reassess  efficacy  ..." 
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CDSS-EBM  Pain  Management  Module 


Please,  choose  option 

Screening  for  pain 

:  °  Reassess  efficacy  for  previously  seen  the  patic 

Figure  37 

>nt 

On  figure  38,  select  "Yes"  option.  Figure  39  will  appear. 

CDSS-EBM  Pain  Management  Module 


Screening  for  pain 

Do  you  have  pain? 

:®Yts 

©No 

Figure  38 

Let's  choose  severe  pain  option  (pain  intensity  9),  and  click  "Submit"  button. 


CDSS-EBM  Pain  Management  Module 


On  scale  0  to  10,  how  would  you  describe  your  pain? 

(Doctor  can  choose  numerical  or  face  description) 

null  5  one  two  ©  three  four  five  six 

seven  eigh^  \ine 

ten 

0  =  no  pain:  1-3  =  mild  pain;  4-6  =  moderate  pain;  7-10 

severe  pain. 1 

(^^Submit 

fll  KCCK  Aflult  CuKg  Pur-  V.1.2010;  pij«  Pur.-A.  1  ofl. 


Figure  39 


Submit 


Next  screen  will  show  up.  Let's  assume  that  pain  increase  from  last  visit.  Doctor  has  to 
select  "Increase"  option. 
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CDSS-EBM  Pain  Management  Module 


Severe  pain 

Compare  with  last  visit,  did  your  pain  score: 

^©^Increased 

Lnchanged 

Decreased 

Figure  40 

And  because  the  patient  is  previously  seen,  let's  assume  that  patient  is  "Opioid  tolerant 
patient". 


CDSS-EBM  Pain  Management  Module 


Pain  level-severe  &  pain  intesity-increase 

What  is  the  patient  status?  ( 

Opioid  naive  patient 

Opioid  tolerant  patient 

Figure  41 

After  making  selection  on  figure  41,  next  screen  will  appear: 

CDSS-EBM  Pain  Management  Module 


Patient  status-opioid  tolerant  &  pain  level-severe  &  pain  intesity-increase 

Current  drug  administration 

Short  acting  opioids  -  routes: 

^  gablets  (peak  effect  in  60  minutes) 

Oral  solution  (peak  effect  in  60  minutes) 

Intravenous  (peak  effect  in  1 5  minutes) 

Full  list  of  pain  medications  (all  routes  in  one  table) 

©  Prescribe  NSAID  for  mild  pain 

Prescribe  weak  agonists  for  moderate  pain 

Prescribe  strong  agonists  for  severe  pain 

Figure  42 

There  are  several  options,  but  let's  for  beginning  continue  with  prescribing  short  acting 
opioid  in  tablets.  Figure  43  will  appear: 
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CDSS-EBM  Pain  Management  Module 


Medication:  (Opioid  tablets) 

Available  dosage: 

▼1  sfce  opioid  equivalency 

* 

< _ " _  | 

Choose  a  total  daily  dose:  ▼  ^ 

1  day  supply: 

Single  dose: 

7  day  supply:  ® 

30  day  supply: 

If  single  dove  required  "crushed”  tablets,  program  will  set  lower  dose  available  with  tablet  size. 

First  step:  Increase  dose  50  -  100  % 

Second  step:  After  2  -  3  cycles  (dose  adjustment),  consider: 


IV  titration 

Treatment  evaluation  and  management  for  severe  pain 


After  generating  prescription,  click  here  for  more  prescribing  option,  or  go  back  to  home  page 


Figure  43 


Let's  assume  that  the  patient  previously  was  receiving  total  120  mg/day.  First  step 
would  be  to  increase  dose  50%  (choose  option  like  on  figure  44)  and  hit  "Generate 
Prescription"  button. 


CDSS-EBM  Pain  Management  Module 


Medication:  (Opioid  tablets) 

Available  dosage: 

Morphine  Sulfate  [▼!  see  opioid  equivalency 

Tablets:  15,  30  mg  IR,  Typical  * 

initial  dose  is  15-60  mg,  q3-4h  as 

Choose  a  total  daily  dose:  180  mg  [_▼] 

1  day  supply:  6  (six)  tablets 

Single  dose:  30  mg  tabs 

7  day  supply:  42  (forty  two)  tablets  ° 

Take  single  tablet  every  4  hours,  as  needed 

30  day  supply:  180  (one  hundred  eighty)  tablets 

Generate  Prescription  If  single  dove  required  "crashed"  tablets,  program  will  set  lower  dose  available  with  tablet  size. 

First  step:  Increase  dose  50  -  100  % 

Second  step:  After  2-3  cycles  (dose  adjustment),  consider: 

IV  titration 

Treatment  evaluation  and  management  for  severe  pain 

After  generating  prescription,  click  here  for  more  prescribing  option,  or  go  back  to  home  page 


Figure  44 
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Prescription  screen  will  appear.  Explanation  about  this  screen  you  can  find  on  page  15 
(after  figure  17) 


If  after  several  dose  adjustments  we  still  don't  have  successful  results,  we  need  to 
choose  "IV  titration"  (shown  on  figure  46).  Choosing  this  option,  figure  47  will  appear. 


CDSS-EBM  Pain  Management  Module 


Medication:  (Opioid  tablets) 

Available  dosage: 

[▼1  see  opioid  equivalence- 

Choose  a  total  daily  dose:  ▼ 

1  day  supply: 

Single  dose: 

7  day  supply: 

30  day  supply: 

If  single  dose  required  "crashed”  tablets,  program  will  set  lower  dose  available  with  tablet  size. 

First  step:  Increase  dose  50  -  100  % 

Second  step:  After  2  -  3  cycles  (dose  adjustment),  consider: 


titration 

Treatment  evaluation  and  management  for  severe  pain 


After  generating  prescription,  dick  here  for  more  prescribing  option,  or  go  back  to  home  page 


Figure  46 
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CDSS-EBM  Pain  Management  Module 


Calculation  tvis  uit  according  to  StarJet  University  Medical  School  ar.d  Medical  College  ofWiscoasiii 

After  generating  order  form,  click  here  for  more  prescribing  option,  or  go  back  to  home  page 


Figure  47 


Let's  choose  Morphine  for  IV  medication.  Next  screen  will  appear: 


CDSS-EBM  Pain  Management  Module 


Intravenous  (TV)  Patient  Controlled  Analgesia  (PCA) 


Important  explanation! 


Opioid  Morphine  (mg/day)  [  ▼  |  see  opioid  equivalency  Total  daily  dose:  24 


day 


IV  PCA  Morphine 
Basal  Rate:  1  mg  h 


Concentration: | ISO  mg/150  ml  NS  (1  mg/ml) Standard Concentrati^nT^I 


Recommended  0  - 1  mg/hour 


On-Demand:  0  25  mg  dose 
Lockout  Interval:  15  minutes 


Loading  Dose: 


Generate  Order  Form 


Recommended  0  5-1  mg/dose 
Recommended  10-15  minutes 


Recommended  2  -  4  mg 


1  University  Medical  School  ar.d  Medical  College  of^'iscoMta 

After  generating  order  form,  click  here  for  more  prescribing  option,  or  go  back  to  home  page 


Figure  48 


There  are  several  possibilities  for  adjustment. 

Total  daily  dose  (1),  concentration  (2),  lockout  interval  (3)  and  loading  dose  (4). 

After  making  all  selections,  hit  "Generate  Order  Form"  button.  Figure  49  will  appear. 
Explanation  for  this  screen  is  very  similar  to  explanation  on  page  16,  after  figure  17. 
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If  we  choose  "Treatment  evaluation  ..."  on  figure  46,  page  29,  next  screen  will  appear. 
There  are  full  list  of  treatment  possibilities  for  opioid  tolerant  patient. 


CDSS-EBM  Pain  Management  Module 


Subsequent  pain  management  and  treatment  in  opioid  tolerant  patient 

For  ALL  pain  levels 

Recognize  and  treat  adv  erse  effects 

O  Co-analgesic  (adjuv  ant  medications) 

Provide  psychosocial  support,  including  patient  and  family  education 

Opioid  conv  ersion 

Severe  Pain  7-10 

See  mangement  for  ALL  levels  of  pain  above,  and 

Rapidly  titrate  short- acting  opioid  with  opition  for  opioids  conversion 

O  Severe  pain,  full  list  of  strong  agonists 

Reevaluate  working  diagnosis  with  a  comprehensive  pain  assessment 

Moderate  Pain  4-6 

See  mangement  for  ALL  levels  of  pain  above,  and 

Rapidly  titrate  short- acting  opioid  with  opition  for  opioids  conversion 
Moderate  pain  medications,  including  weak  agonists 

Reevaluate  working  diagnosis  with  a  comprehensive  pain  assessment 

Mild  Pain  1  -  3 

See  mangement  for  ALL  levels  of  pain  above,  and 

Prescribe  NS  AID  (Modify  regimen  to  minimize  adverse  effects) 

Prescribe  short- acting  opioid  (Pain  is  under  control  with  opioids) 

After  any  treatment.  Reevaluate  patient's  goals  of  comfort  and  function  at  each  contact  Figure  50 
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V.  Pain  intensity  rating 

This  sub-module  contains  the  pain  intensity  rating  scales.  By  selecting  this  option,  the 
user  will  be  guided  through  assessment  of  patient's  pain  intensity. 

Using  NCCN  (National  Comprehensive  Cancer  Network)  guidelines,  we  developed  a 
submodule  for  universal  and  comprehensive  pain  assessment  with  both  numerical  and  the 
faces  pain  rating  scale  detection  (figure  11,  page  14).  Depending  on  the  pain  level,  we  also 
provided  a  suggested  list  with  medications,  dosage,  duration  and  route. 

Pain  intensity  must  be  quantified  by  the  patient  whenever  possible.  According  to  NCCN 
guidelines,  pain  intensity  is  divided  in  the  following  groups: 

•  No  pain  =  0 

•  Mild  pain  =  1-3 

•  Moderate  pain  =  4- 6 

•  Severe  pain  =  7-10 

By  clicking  on  the  radio  button  for  pain  level  or  moving  the  sliding  button  till  we  find  the 
correct  pain  level  and  then  pushing  the  corresponding  "Submit"  button  will  allow  the  program 
to  jump  to  next  page  with  suggested  corresponding  pain  level  medication.  After  choosing  the 
medication,  dosage,  route  and  dispense,  doctor  will  have  the  option  to  directly  print  the 
prescription  and  give  it  to  the  patients  (or  family  member/  friend)  or  to  send  an  e-mail  to  the 
pharmacist  with  the  prescription  data. 

To  access  the  Pain  intensity  sub-module  select  Pain  intensity  rating  as  shown  below. 
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CDSS-EBM  Pain  Management  Module 


What  would  you  like  to  do? 


Pain  Assessment 


Sub-Modules  for  Pain  Assessment 


General  conversion 
Intravenous  protocol 
Transdermal  patch 


Antidepressants 
Anticonvulsants 
Topical  agents 
Corticosteroids 


Constipation  &  Nausea 
Pruritus  &  Delirium 
Others 


Nonopioid  Analgesic 
Opioid  Analgesic 
Co-Analgesic 


Instructions 


D'.sclairrer 


(jpioid  Conversion 

Neuropathic  Pain 

Adverse  Effects 

Information  on  Drugs 

Pain  Intensity  Rating 


Visual  rating  scale 


Figure  51 


This  will  take  you  to  the  next  screen  (figure  52). 

CDSS-EBM  Pain  Management  Module 


Options 

What  is  the  patient  status? 

1  O  Opioid  naive  patient 

Opioid  tolerant  patient 

Figure  52 

Selecting  either  options,  program  will  bring  us  to  pain  intensity  screen  and  from  there  is 
same  procedure  like  was  previously  explained  (figure  11,  page  14). 

VI.  Opioid  Conversion 

This  sub-module  contains  the  opioid  conversion  calculator.  By  selecting  this  option,  the  user 
will  be  able  to  calculate  dosage  conversions  between  different  opioid  drugs  and  routes  of 
administration. 

There  are  different  conversion  types  available: 

•  General  conversion 

•  Between  intravenous  (IV)  medications 

•  Intravenous  (IV)  to  oral  medications  (PO) 
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•  Multiple  PO  and  fentanyl  patch  convert  to  IV  medications 

•  Between  oral  medications  with  rescue  doses 

•  Opioids  to  fentanyl  transdermal  patch  (Duragesic) 

To  access  the  Opioid  conversion  sub-module  select  Opioid  conversion  as  shown  below. 


CDSS-EBM  Pain  Management  Module 
What  would  you  like  to  do? 


Pain  Assessment 


Modules  for  Pain  Assessment 


Adverse  Effects 


information  on  Drugs 


Numerical  ratine  scale 
Visual  ratine  scale 


Disclaimer 


neral  converge 
IntravSW^^olocol 
Transdermal  patch 


Antidepressants 
Anticonvulsants 
Topical  agents 
Corticosteroids 


Constipation  &  Nausea 
Pruritus  &  Delirium 
Others 


Nonopioid  Analgesic 
Opioid  Analgesic 
Co -Analgesic 


Instructions 


Pain  Intensity  Ral 


Opioid  Conversion 


uropathic  Pain 


Figure  53 


By  clicking  "Opioid  Conversion"  button,  next  screen  will  appear: 


CDSS-EBM  Pain  Management  Module 


Conversion  GRADE  evidence  profile 

Select  conversion  option: 

show  bide  references 

General  conversion  *’ 3’ 5 

Between  intravenous  (IV)  medications  6; 

0  Intravenous  (TV)  to  oral  medications  (PO) 

Multiple  PO  and  fentanyl  patch  convert  to  intravenous  n 
Between  oral  medications  with  rescue  doses 

Opioids  to  fentanyl  transdermal  patch  (Duragesic)  4 

ledications 

Figure  54 

Base  of  our  needs,  doctors  can  choose  any  option  by  clicking  corresponding  radio 

button. 
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We  are  encourage  users  to  try  all  different  options  and  "familiarized"  with  program, 
before  actual  need. 


VII.  Neuropathic  Pain 

This  sub-module  contains  four  main  groups: 

•  Antidepressants 

•  Anticonvulsants 

•  Topical  agents 

•  Corticosteroids 

For  all  these  options,  you  need  to  select  "Medication",  and  choose  amount  for  "Total 
daily  dose"  and  program  will  automatically  display  value  for  "Single  dose",  based  on  how  often 
patients  need  to  take  the  medication.  After  this,  the  button  "Generate  Prescription"  will  be 
"alive"  and  doctor  will  be  able  to  generate  prescription. 

To  access  Neuropathic  pain  sub-module,  select  Neuropathic  pain  as  shown  below. 


CDSS-EBM  Pain  Management  Module 
What  would  you  like  to  do? 


Sub-AJ 

Pain  Intensity  Rating 

Opioid  Conversion 

Neuropathic  Pain 

Adverse  Effects 

Information  on  Drugs 

Numerical  ratine  scale 
Visual  ratine  scale 


General  conversion 
Intravenous  protocol 
Transdermal  patch 


Antidepressants 

Topical  agents 
Corticosteroids 


Constipation  &  Nausea 
Pruritus  &  Delirium 

Others  _ 


Nonopioid  Analgesic 
Opioid  Analgesic 
_ C  o  -  Analgesic 


Figure  55 


Next  screen  will  show  up: 
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CDSS-EBM  Pain  Management  Module 


Co-analgesics  (adjuvant)  medications  for  neuropathic  pain 


Select  co-analgesic  (adjuvant)  medications 

see  principles 


Anticomulsants 
Topical  agents 
O  Corticosteroids 


lepressants 


Figure  56 


By  choosing  "Antidepressants"  option,  figure  22  from  page  18  will  appear.  Explanation 
and  how  to  continue  was  explained  on  page  18  -  21. 

VIII.  Adverse  Effects 

The  most  common  types  of  adverse  effects: 

•  Constipation 

•  Nausea  and  Vomiting 

•  Pruritus 

•  Delirium 

•  Motor  and  Cognitive  Impairment 

•  Respiratory  depression  (The  most  serious  Adverse  effect) 

•  Sedation 

Generating  prescription  procedure  is  similar  to  procedure  for  Neuropathic  pain. 

To  access  adverse  effects  sub-module,  select  adverse  effects  as  shown  below. 
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CDSS-EBM  Pain  Management  Module 


What  would  you  like  to  do? 


By  clicking  "Adverse  effects"  button,  next  screen  will  appear: 

CDSS-EBM  Pain  Management  Module 


The  most  common  types  of  adverse  effects 

Select  adverse  effect  which  you  would  like  to 
treat 

see  principles 

Constipation 

Nausea  and  Vomiting 

C  Pruritus 

Delirium 

t  Motor  and  cognitive  impairment 

_  Respirators'  depression 

Sedation  '  Picture  58 

Base  of  the  patient  needs,  doctors  can  choose  any  option  by  clicking  corresponding 
radio  button. 

We  are  encourage  users  to  try  all  different  options  and  "familiarized"  with  program, 
before  actual  need. 


IX.  Information  on  Drugs 

There  are  three  main  groups  with  corresponding  sub-groups: 
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•  Nonopioid  Analgesic 
-Acetaminophen  (Tylenol) 

-  Salicylate 

-  Non  Selective  NSAIDs  (Non-steroidal  anti-inflammatory  drugs) 

-  Selective  COX-2  Inhibitor 

•  Opioid  Analgesic 

-  Strong  full  Agonists 

-  Weak  full  Agonists 

-Weak Agonist/Reuptake  Inhibitor 

-  Partial  Agonist  and  Mixed  Agonist/Antagonists 

•  Co-Analgesic  (Adjuvant  medication)  for  neuropathic  pain 

-  Antidepressants 

-  Anticonvulsants 

-  Other  Adjuvant 

From  each  product  insert  for  each  of  the  most  commonly  used  medications,  we 
extracted  data  on  dosage,  cycles,  route,  duration  of  the  medications,  time  to  peak  effect,  and 
initial  dosage.  We  then  presented  a  summary  on  a  separate  table  for  a  quick  reference.  On  the 
same  table,  we  also  provided  a  link  for  each  product  insert  for  a  more  detailed  explanation. 

Drugs  information  sub-module  is  short  cuts  to  explanation  about  dosage,  routes  and 
other  useful  information  about  drugs  with  option  to  access  package  insert. 

To  access  drugs  information  sub-module,  select  information  on  drugs  as  shown  below. 
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CDSS-EBM  Pain  Management  Module 
What  would  you  like  to  do? 


Visual  ratine  scale 


Intravenous  protocol 
Transdermal  patch 


Anticonvulsants 
Topical  agents 
Corticosteroids 


Pruritus  &  Delirium 
Others. - 


Dtscliimg 


Opioid  Analgesic 
'o-Analgesic 


Figure  59 


Instructions 


By  clicking  "Information  on  drugs"  button,  next  screen  will  appear: 

CDSS-EBM  Pain  Management  Module 


Drugs 


Information  on  drugs? 


Figure  60 


Nonopioid  analgesic  dosages  and  product  insert  for  these  medications 
ioid  analgesic  dosages  and  product  insert  for  these  medications 
Co-analgesic  dosages  and  product  insert  for  these  medications 


Let's  for  explanation  purpose,  click  on  radio  button  for  "Opioid  analgesic  ..." 
Screen  bellow  will  appear: 
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There  are  short  extract  data  for  dosage  form,  duration,  half-life,  time  to  peak  effect  and 
suggested  starting  dose.  By  clicking  on  highlight  medication,  program  will  move  to  short 
explanation  about  that  drug  with  option  to  select  and  original  package  insert  for  that 
medication. 


IX.  Write  Rx 

If  you  want  only  to  write  a  prescription,  the  fastest  way  is  select  "Write  Rx"  from  menu 

option. 

Clicking  on  "Write  Rx"  button,  on  next  page  you  will  have  the  choice  to  choose  manual 
writing  or  options  with  suggested  dosage  (for  ongoing  care). 


40 


Example  A:  Converting  IV  morphine  to  IV  hydromorphone 

Patient  is  taking  IV  morphine  at  6  mg/h  and  needs  to  be  converted  to  IV 
hydromorphone. 

To  access  IV  conversion,  select  Opioid  conversion  sub-module  as  shown  below. 

CDSS-EBM  Pain  Management  Module 
What  would  you  like  to  do? 


odules  for  Pain  Assessment 


Pain  Intensity  Ralng 

Opioid  Conversion 

isfturopathic  Pain 

Adverse  Effects 

Information  on  Drugs 

Numerical  rat  ins  scale 
Visual  rat  ins  scale 

^^^^General  convert  101^^^^ 

Transdermal  patch 

Antidepressants  Constipation  &  Nausea  Nonopioid  Analgesic 

Anticonvulsants  Pruritus  &  Delirium  Opioid  Analgesic 

Topical  agents  Others  Co-Analgesic 

Corticosteroids 


Disclaimer 


On  next  page,  select  "Between  intravenous  (IV)  medications" 


CDSS-EBM  Pain  Management  Module 


Conversion  GRADE  evidence  profile 


General  conversion  *’ 3  ' 

( 

^^Between  intravenous  (TV")  medications 

Select  conversion  option: 

Intravenous  (TV)  to  oral  medications  (PO) 

show  hide  r efer enc  es 

Multiple  PO  and  fentanyl  patch  convert  to  intravenous  medications 

O  Between  oral  medications  with  rescue  doses 

Opioids  to  fentanyl  transdermal  patch  (Duragesic)  4 

By  clicking  on  radio  button,  program  will  bring  us  to  the  next  page: 
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CDSS-EBM  Pain  Management  Module 


Current  intravenous  (TV)  medications: 

- - - 

New  intravenous  (H7)  medications: 

Opioid  OZO 

Opioid 

G 

Current  daily  dose:  mg  day 

New  daily  dose: 

mg  day  Calculate 

Intravenous  (IV)  Patient  Controlled  Analgesia  (PCA)  see  opioid  equivalency 


IV  PCA  jr]  Concentration:  j 

Basal  Rate:  mg  h  Recommended 

On -Demand:  mgdose  Recommended 

Lockout  Interval:  Recommended 

Loading  Dose:  ▼  Recommended 

If  Panaratfl  OrHor  Pnrm  r 

Calculation  v,is  sade  according  to  Stanford  University  Medical  School  and  Medical  College  ofWisconsin 

Make  selection  for  current  IV  medication  (morphine)  and  new  IV  medication 
(hydromorphone). 


CDSS-EBM  Pain  Management  Module 


Calculation  was  sade  according  to  Stanford  University  Medical  School  and  Medical  College  ofWisconsin 

Program  require  daily  dose  for  current  IV  medication. 
(6  mg/h  *  24  h  =  144  mg/day) 

Let's  enter  value  in  table. 
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CDSS-EBM  Pain  Management  Module 


Current  intravenous  (IV)  medications: 

New  intravenous  (TV)  medications: 

Opioid  Morphine  (mg/day)  [▼] 

Opioid  Hydromorphone  (mg/day)  Q 

Current  daily  dose:  144|  mg  day 

New  daily  dose:  mg  day  ^  Calculate 

Intravenous  (TV)  Patient  Controlled  Analgesia  (PCA)  see  opioid  equivalency 


IYPCA  Hydromorphone  (Oilaudid)  [3]  Concentration  [▼] 
Basal  Rate:  mgh  Recommended 

On-Demand:  mgdose  Recommended 

Lockout  Interval:  ▼  Recommended 

Loading  Dose:  Recommended 


Calculation  v.u  accccdif.?  to  StarJord  University  Medical  School  aid  Medical  Co!!*?*  ofWiscor.sin 


After  entering  value  for  Current  daily  dose,  hit  "Calculate"  button. 


CDSS-EBM  Pain  Management  Module 


Current  intravenous  (TV)  medications: 

New  intravenous  (IV)  medications: 

Opioid  Morphine  (mg/day)  [▼] 

Opioid  Hydromorphone  (mg/day)  Q 

Current  daily  dose:  144  mg  day 

New  daily  dose:  21  599999  mg  day 

psai 

Intravenous  (TV)  Patient  Controlled  Analgesia  (PCA) 


‘  opioid  equh 


IV  PCA  Hydromorphone  (Oilaudid)  [^]  Concentratioa  30  mg/150  ml  NS  (0  2  mg/ml)  Standard  Concentration  ^]l 
Basal  Rate:  0  9 


mgh 


On-Demand  0  225  mg  dose 


Recommended  0-02  mg/hour 
Recommended  0  1-02  mg/dose 


Lockout  Interval: 

15  minut 

es!^ 

2 

Recommended  10-15  minutes 

Loading  Dose: 

0  4  mg-- 

3 

Recommended  0.4  -  0.8  mg 

Generate  Order  Form 


-try  University  Medical  School  *r.£  Medical  Coll*?*  cfWisconsin. 


Before  we  generate  order  form,  there  are  several  possibilities  for  adjustment: 
Concentration  (1),  lockout  interval  (2)  and  loading  dose  (3). 

After  making  all  selections,  hit  "Generate  Order  Form"  button.  Next  screen  will  appear. 
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Intravenous  (TV)  Patient  Controlled  Analgesia  (PCA)  -  Standardized  Order  Form 


USF 

12901  Bruce  B.  Dor.™  Blvd.,  MDC  33 
Tampa.  FL  33612 


Tuesday,  6/7/2011  11:33  AM 


Name: 


IV  PCA:  Hydromorphone  (Dilaudid) 

30  mg/150  ml  NS  (0.2  mg/ml)  Standard  Concentration 

Basal  rate:  0.9  mg/hour 

On-Demand  amount:  0.225  mg/dose 

Lockout  interval:  15  minutes 

Loading  dose:  0.4  mg 


Phone: 


ME# 


DEA= 


Send  order  form  to  Pharmacist 


Print  this  page 
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Example  B:  Converting  oral  Oxymorphone  to  transdermal  fentanyl  patch 

Patient  is  taking  15  mg  every  12  hours  and  needs  to  be  converted  to  transdermal 
fentanyl  patch. 

Select  Opioid  conversion  sub-module  as  shown  below: 

CDSS-EBM  Pain  Management  Module 
What  would  you  like  to  do? 


lodules  for  Pain  Assessment 


Pain  Intensity  Ralng 

Opioid  Conversion 

Nluropathic  Pain 

Adverse  Effects 

Information  on  Drugs 

Numerical  rat  ins  scale 
Visual  rat  ins  scale 

^^^General  conversion^^^ 

Transdermal  patch 

^  Antidepressants  Constipation  &  Nausea  Nonopioid  Analsesic 

Anticonvulsants  Pruritus  &  Delirium  Opioid  Analgesic 

Topical  agents  Others  Co-Analgesic 

Corticosteroid* 


Disclaimer 


On  next  page,  select  "Opioids  to  fentanyl  transdermal  patch  (Duragesic)" 


CDSS-EBM  Pain  Management  Module 


Conversion  GRADE  evidence  profile 


General  conversion  3’ 5 

•  2  6  7 

Between  intravenous  (TV)  medications  * 

Select  conversion  option: 

O  Intravenous  (TV7)  to  oral  medications  (PO) 

show  hide  references 

Multiple  PO  and  fentanyl  patch  convert  to  intravenous  medications 

Between  oral  medications  with  rescue  doses 

( 

^^)Opioids  to  fentanyl  transdermal  patch  (Duragesic)  4 

By  clicking  on  radio  button,  program  will  bring  us  to  the  next  page: 
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CDSS-EBM  Pain  Management  Module 


Current  medications: 

Important  explanation! 

\ 

Opioid  (  ▼!  ) 

Total  daily  dose:  mg  day  Calculate 

V  S 

Fentanvl  transdermal  patch  (Duragesic): 

Available  Dose:  25.  50.  75.  100  mcg/h  transdermal  patches 

Opioid  Fentanyl  (Duragesic) 

Transdermal  patch: 

7  day  supply:  @ 

30  day  supply: 

show/hide  Recommended 

’  All  Conversion  v.-as  rz-te  according  to  pacScage  insert  fcr  Duagesic  That  is  dtfeent  then  NCCN  guidelines  for  Adult  Cancer  Pain 


Make  selection  for  current  medication  (Oxymorphone) 

CDSS-EBM  Pain  Management  Module 


Current  medications: 

Important  explanation! 

Opioid  Oxymorphone  (oral)  [▼] 

Total  daily  dose:  mg  day  Calculate 

Fentanvl  transdermal  patch  (Duragesic): 

Available  Dose:  25.  50.  75.  100  mcg/h  transdermal  patches 

Opioid  Fentanyl  (Duragesic) 

Transdermal  patch: 

7  day  supply:  % 

30  day  supply: 

show  hide  Recommended 

All  Conversion  xvas  aade  according  to  package  insert  for  Duragesic  That  is  difeer.t  then  NCCN*  guidelines  for  Adult  Cancer  Pain 


Program  require  daily  dose  for  current  medication. 
(15  mg  *  2  =  30  mg/day) 

Let's  enter  value  in  table. 
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CDSS-EBM  Pain  Management  Module 


Current  medications: 

Important  explanation! 

Opioid:  Oxymorphone  (oral)  [▼] 

Total  daily  dose:  30  mg  day  ^^^Calculate  ] 

Fentanvl  transdermal  patch  (Duragesic): 

Available  Dose:  25.  50.  75.  100  meg/h  transdermalpalches 

Opioid:  Fentanyl  (Duragesic) 

Transdermal  patch: 

7  day  supply:  # 

30  day  supply: 

show  hide  Recommended 

All  Conversion  tvis  according  to  piciai*  insert  fee  Daragesic  That  is  difeent  then  NCCN  guidelines  fcr  Adult  Cancer  Pain. 


After  entering  value  for  Current  daily  dose,  hit  "Calculate"  button. 


CDSS-EBM  Pain  Management  Module 


Current  medications: 

Important  explanation! 

Opioid:  Oxymorphone  (oral)  [▼] 

Total  dailv  dose:  30  mg  day  \  Calculate  ] 

Fentanvl  transdermal  patch  (Duragesic): 

Available  Dose:  25.  50.  75.  100  meg/h  transdermal  patches 

Opioid:  Fentanyl  (Duragesic) 

Transdermal  patch:  50  meg/h;  Applied  to  intact  non-irrita 

7  day  supply:  3  (three)  patches  ° 

30  day  supply.  10  (ten)  patches 

Generate  Prescription  j 

Replace  patch  every  72  hours  show  hide  Recommended 

Hit  "Generate  Prescription"  button.  Next  screen  will  appear. 


USF 

fSVaKffiVn 

12901  Bruce  B.  Down*  Blvd.,  MDC  33 
Tampa,  FL  33612 

Phone: 

>  j  ME  # 


Name:  Tuesday,  June  07,  2011 

Rx:  Fentanyl  (Duragesic),  Single  dose:  50  meg/h;  Applied  to  intact  non-irritated  skin;  Each  time 
applied  to  different  skin  site. 

Dosage:  Replace  patch  every  72  hours 
Dispense:  Seven  day  supply,  3  (three)  patches 


Refill:  0=E  1  =  0  2=0  3=0  4=0  PRN=0 


DEA« 


Send  prescription  to  Pharmacist 


|  Print  this  page 
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Example  C  Converting  multiple  opioid  to  intravenous  medication 


Patient  is  taking  90  mg  morphine  Avinza  (extended  release)  every  24  hours,  5  mg 
Oxycodone  (Oxy  IR)  every  6  hours  and  transdermal  fentanyl  patch  50  mcg/hours.  Because  he  is 
still  in  severe  pain,  he  was  admitted  in  hospital  and  doctor  wants  to  convert  all  medication  in 
intravenous  morphine. 

Select  Opioid  conversion  sub-module  as  shown  below: 


CDSS-EBM  Pain  Management  Module 
What  would  you  like  to  do? 


lodules  for  Pain  Assessment 


Pain  Intensity  Rafng 

Opioid  Conversion 

Neuropathic  Pain 

Adverse  Effects 

Information  on  Drugs 

Numerical  rating  scale 
Visual  rating  scale 

^^^^er.-rral  conversic^^^^ 

Transdermal  patch 

^  .Antidepressants  Constipation  i:  Nausea  Nonopioid  Analgesic 

Anticonvulsants  Pruritus  &  Delirium  Opioid  Analgesic 

Topical  agents  Others  Co- Analgesic 

Corticosteroids 


BgwmMi 


On  next  page,  select  "Multiple  PO  and  fentanyl  patch  convert  to  intravenous 
medication" 


CDSS-EBM  Pain  Management  Module 


Conversion  GRADE  evidence  profile 

Select  conversion  option: 

show  hide  references  ( 

General  conversion  *’ 3?  5 

•  •  •  2  6  7 

©  Between  intravenous  (TV7)  medications  '  * 

^Intravenous  (IV)  to  oral  medications  (PO) 

^)vlultiple  PO  and  fentanyl  patch  convert  to  intravenous  medications 
Between  oral  medications  with  rescue  doses 

Opioids  to  fentanyl  transdermal  patch  (Duragesic)  4 

By  clicking  on  radio  button,  program  will  bring  us  to  the  next  page: 
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CDSS-EBM  Pain  Management  Module 


Current  medications: 

Oral  opioid  1  ▼  j 

Daily  dose:  mg  day 

Oral  opioid  2  [▼] 

Daily  dose:  mg  day 

Transdermal  patch  ▼] 

Daily  dose:  mg  day 

New  intravenous  (IV)  medications:  ▼ 

Equivalent  morphine  oral  dose: 

mg  day  New  IV  daily  dose:  mg  day  Calculate 

Intravenous  (IV)  Patient  Controlled  Analgesia  (PCA) 

IV  PCA  0 

Concentration:  ▼ 

Basal  Rate:  mg  h 

Recommended 

On-Demand  mgdose 

Recommended 

Loading  Dose:  ▼  J 

Recommended 

Lockout  Interval  ▼  j 

Recommended 

*  Product  insert  fcr  Durag«s:c  (F*f.tartyl  trsr.sderzi!  patch)  us<  conservative  conversion  sethod:  60cg  PO  aorphir.e  equal  ICtsg  I\'  aorphine 


Program  require  daily  dose  for  current  medication. 
Morphine  Avinza:  (90  mg/day) 

Oxycodone  Oxy  IR:  (5  mg  *  4  =  20  mg/day) 

Fentanyl  patch:  (50  mcg/h  *  24  h  =  1.2  mg/day) 

Let's  enter  values  in  table  and  select  new  IV  medication. 
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CDSS-EBM  Pain  Management  Module 


Current  medications: 

Oral  opioid  1 

Morphine  ▼! 

Daily  dose:  90  mg  day 

Oral  opioid  2  Oxycodone  ▼] 

Daily  dose:  20  mg  day 

Transdermal  patch  Fentanyl  [▼]  One  patch  25mcg/h  =  0  6 mg/d; 


Daily  dose:  1.2 


mg  day 


New  intravenous  (TV)  medications:  ! 


s' 

> 

Equivalent  morphine  oral  dose:  mg  day 

New  IV  daily  dose:  mg  day  ( 

Calculate 

Intravenous  (TX7)  Patient  Controlled  Analgesia  (PCA) 

IV  PCA  Morphine  Q  Concentration:  150  mg/150  ml  NS  (1  mg/ml) 

Basal  Rate:  mg  h  Recommended  0  -  1  mg/hour 

On-Demand:  mg  dose 


Recommended  0.5-1  mg/dose 


Loading  Dose:  2  mg 
Lockout  Intend  15  minutes  [▼] 


Recommended  2  -  4  mg 


Recommended  10-15  minutes 


"  Product  ir.s«ft  fcr  Dungesic  (Fer.tar.yl  trar.sdsrxil  patch)  uh  coeutrvitivt  converaiort  xsthod:  tOxz  PO  sorphir.*  «qual  10s:g  I\’  sofphir.e 


After  selecting  opioids  and  entering  value  for  daily  dose,  hit  "Calculate"  button. 


CDSS-EBM  Pain  Management  Module 


Current  medications: 

Oral  opioid  1  Morphine 

Daily  dose:  90  mg  day 

Oral  opioid  2  Oxycodone  ^ 

Daily  dose:  20  mg  day 

Transdermal  patch  Fentanyl  [▼ 

One  patch  25mcg/h  =  0  6 mg/d;  Daily  dose:  1.2  mg  day 

New  intravenous  (TV)  medications:  Morphine  ▼ 

Equivalent  morphine  oral  dose:  300 

mg  day  New  IV  dailv  dose:  100  mg  day  j  Calculate  ] 

Intravenous  (TX7)  Patient  Controlled  Analgesia  (PCA) 

IV  PCA  Morphine  ) 

Concentration:  150  mg/150  ml  NS  (1  mg/ml)  ] 

Basal  Rate:  4  167  mg  h 

Recommended  0  -  1  mg/hour 

On-Demand:  1  042  mg  dose 

Recommended  0  5-1  mg/dose 

Loading  Dose:  2  mg  ▼] 

Recommended  2  -  4  mg 

Lockout  Interval:  15  minutes  [  ▼  [ 

| Recommended  10  -  15  minutes  C  |  Generate  Order  Form^^) 

*  Product  invert  fcr  Duraeesic  (F«r.tar.y!  trar.sderrrai  patch)  uve  coeturvitivt  conversion  -method  fCxg  PO  aorphine  equal  1  Crcg  I\raorp7!tr»- 
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Let's  explain  calculation: 


From  opioid  conversion  guidelines,  20  mg  oral  Oxycodone  is  equal  to  30  mg  Morphine. 
From  package  insert  for  Duragesic  (transdermal  fentanyl  patch)  50mcg/h  is  equal  to 
(135  +224)/2  =  179.5  mg/day  oral  morphine. 

Equivalent  oral  Morphine  dose  is:  90  +  30  +  180  =  300  mg/day 
Parenteral  morphine  is  1/3  oral  morphine:  300/  3  =  100  mg/day 
Hit  "Generate  Order  Form"  button  on  page  46,  next  screen  will  appear. 


Intravenous  (IV)  Patient  Controlled  Analgesia  (PCA)  -  Standardized  Order  Form 


Tuesday,  6/7/2011  1:37  PM 


12901  Bruce  B.  Dov.tis  Blvd..  MDC  33 
Tampau  FL  33612 


Name: 


IV  PCA:  Morphine 

150  mg/150  ml  NS  (1  mg  ml) 

Basal  rate:  4.167  mg/hour 
On-Demand  amount:  1.042  mg/dose 
Lockout  interval:  15  minutes 
Loading  dose:  2  mg 


0 


Phone: 


ME* 


DEA? 


Send  order  form  to  Pharmacist 


Print  this  page 
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Appendix  14  EB-PMM  Evaluation  Survey 


8/27/12 


Qualtrics  Survey  Software 


Default  Question  Block 


I  am  a 

©  Physician 
Q  Physician  Assistant 
0  Registered  nurse 
0  Other  (please  specify) 


My  area  of  practice  is 

Family  medicine 
|r  Hematology 
Oncology 
Palliative  medicine 
Other  (please  specify) 


Typically,  I  make  decisions  regarding  pain  management 


times  per  day.  (Please,  indicate  number  of  times) 


In  my  practice,  I  currently  use  the  following  forms  of  clinical  decision  support 


Yes 

No 

Paper  (i.e.  guidelines) 

© 

© 

Computer  or  Internet  based 
information  (i.e.  on-line  calculators) 

0 

© 

Clinical  decision  support  system 
(incorporated  into  institutional 

EMR) 

© 

© 

1  do  not  use  any  form  of  clinical 
decision  support 

0 

© 

Other  (please  specify) 

© 

© 

The  following  statements  refer  to  perceived  usefulness  of  EB-PMM  if  it  were  made  available  in  practice.  Please,  indicate  the 
extent  to  which  you  agree/disagree  with  each  statement. 


Disagree 

strongly 

Disagree  Neutral 

Agree  Agree  strongly 

Not  applicable 

Enhance  my  efficiency  of 
prescribing 

© 

©  © 

© 

© 

© 

Improve  my  productivity 

© 

©  © 

© 

© 

© 

Improve  the  quality  of  care  1  can 
provide 

© 

©  © 

© 

© 

© 

Make  my  job  easier 

© 

©  © 

© 

© 

© 

The  following  statements  refer  to  perceived  functions  of  EB-PMM.  Please  indicate  the  extent  to  which  you 
agree/disagree  with  each  statement. 

Disagree  strongly 

Disagree 

Neutral 

Agree 

Agree  strongly 

To  evaluate  patients’  pain  intensity 

© 

© 

© 

© 

© 

To  make  decision  regarding  choice 
of  treatment 

© 

© 

© 

0 

© 

To  perform  opioid  conversion 
calculations 

© 

© 

© 

© 

© 
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To  obtain  information  on  drugs 
used  for  pain  management  from 

0 

0 

0 

0 

0 

package  insert 

To  obtain  information  on  treatment 
related  adverse  events 

0 

0 

0 

0 

0 

To  obtain  evidence  on  drugs  used 
in  pain  management 

0 

0 

0 

0 

0 

To  write  prescriptions 

0 

0 

0 

0 

0 

To  perform  other  tasks  (Please 
specify  below) 

0 

0 

0 

0 

0 

I  would  like  to  see  the  following  functions  added  to  EB-PMM.  (Please  specify) 


A 


I  would  like  to  see  the  following  functions  already  in  EB-PMM  further  developed.  (Please  specify  the  functions) 


A 


The  following  statement  refers  to  perceived  quality  of  EB-PMM.  Please  select  the  number  which  most  appropriately 
reflects  your  impressions  of  using  EB-PMM.  If  you  have  not  had  experience  with  certain  elements,  please  select 
“  not  applicable”  (NA). 


Hard  to  read  Easy  to  read  Not 

Applicable 

0123456789  10 


Characters  on  screen 

(font,  colors, 
arrangement)  are 

□ 

Inadequate  Adequate  Not 

Applicable 

0123456789  10 


Amount  of  information 
displayed  on  screens 

□ 

is 

Confusing  Clear  Not 

Applicable 

0123456789  10 
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sequence  ot  screens 
for  any  specific  task 

□ 

is 

Inconsistent  Consistent  Not 

Applicable 

0123456789  10 


Use  of  medical 

terminology 

□ 

throughout  EB-PMM 

is 

Confusing  Clear  Not 

Applicable 

0123456789  10 


On-screen 

instructions  for 

□ 

commands  or 

functions  are 

Impossible  Easy  Not 

Applicable 

0123456789  10 


Finding  solution  to 
problem  in  technical 

□ 

manual  is 

Too  many  Just  right  Not 

Applicable 

0123456789  10 


Number  of  steps 

□ 

needed  to  complete 

tasks  are 

Never  Always  Not 

Applicable 

0123456789  10 


Steps  to  complete 
tasks  follow  logical 

□ 

sequence 
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L 


Difficult  Easy  ^ot 

Applicable 

0123456789  10 


Ability  to  correct 

mistakes  (typos, 

□ 

incorrect  selections) 

is 

Poor  Excellent  Not 

Applicable 

0123456789  10 


The  overall  quality  of 
the  EB-PMM  interface 

n 

is 

I  have  the  following  additional  comments  regarding  EB-PMM  interface  quality:  (Please  specify) 


A 


The  following  statements  refer  to  your  general  impression  about  the  EB-PMM.  Please,  indicate  the  extent  to  which  you 
agree/disagree  with  each  statement. 


Disagree  strongly 

Disagree 

Neutral 

Agree 

Agree  strongly 

1  felt  very  confident  using  EB-PMM 

0 

0 

0 

0 

0 

1  found  that  various  functions  of 
EB-PMM  were  well  integrated 

0 

0 

0 

0 

0 

1  thought  the  system  was  easy  to 
use 

0 

0 

0 

0 

0 

1  would  use  EB-PMM  frequently  if 
it  was  made  available  in  practice 

0 

0 

0 

0 

0 

Overall,  1  am  satisfied  with  the  EB-PMM 

Disagree  strongly 

Disagree 

Neutral 

Agree 

Agree  strongly 

© 

0 

0 

0 

0 

I  have  the  following  additional  comments  I  would  like  to  make  regarding  EB-PMM: 
(Please  specify) 
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